[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:04 dist-test-slave-10pc krb5kdc[30824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:04 dist-test-slave-10pc krb5kdc[30824](info): set up 2 sockets
Aug 06 03:42:04 dist-test-slave-10pc krb5kdc[30824](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 03:42:05.502892 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:05 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:05.629810 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35967
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:05.717366 30837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:05.717574 30837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:05.717598 30837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:05.718959 30837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:05.719022 30837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:05.719039 30837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:05.719053 30837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:05.719065 30837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:05.720849 30837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:35967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:05.721246 30837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:05.721565 30837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:05.727707 30843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:05.727732 30845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:05.727972 30837 server_base.cc:1034] running on GCE node
W20260806 03:42:05.729784 30842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:05.730130 30837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:05.730540 30837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:05.731716 30837 hybrid_clock.cc:648] HybridClock initialized: now 1785987725731696 us; error 43 us; skew 500 ppm
Aug 06 03:42:05 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:05.734395 30837 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:05.735108 30837 webserver.cc:460] Webserver started at http://127.30.24.62:36227/ using document root <none> and password file <none>
I20260806 03:42:05.735452 30837 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:05.735530 30837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:05.735720 30837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:05.737087 30837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "36b2dc5e421841ba8838339449038f59"
format_stamp: "Formatted at 2026-08-06 03:42:05 on dist-test-slave-10pc"
server_key: "e638fae4f81e153915112afbe2f21a04"
server_key_iv: "293620243f92cb51729d922c095efe0f"
server_key_version: "encryptionkey@0"
I20260806 03:42:05.737520 30837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "36b2dc5e421841ba8838339449038f59"
format_stamp: "Formatted at 2026-08-06 03:42:05 on dist-test-slave-10pc"
server_key: "e638fae4f81e153915112afbe2f21a04"
server_key_iv: "293620243f92cb51729d922c095efe0f"
server_key_version: "encryptionkey@0"
I20260806 03:42:05.739475 30837 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:05.740761 30852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:05.741104 30837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 03:42:05.741245 30837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "36b2dc5e421841ba8838339449038f59"
format_stamp: "Formatted at 2026-08-06 03:42:05 on dist-test-slave-10pc"
server_key: "e638fae4f81e153915112afbe2f21a04"
server_key_iv: "293620243f92cb51729d922c095efe0f"
server_key_version: "encryptionkey@0"
I20260806 03:42:05.741330 30837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:05.764115 30837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:05.766448 30837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:05.766659 30837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:05.771708 30837 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:35967
I20260806 03:42:05.771801 30905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:35967 every 8 connection(s)
I20260806 03:42:05.772399 30837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:05.773967 30906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:05.775789 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 30837
I20260806 03:42:05.775915 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:05.776598 30816 external_mini_cluster.cc:1428] Setting key cc12d0ced2343f133f3b00d1c8d8302e
I20260806 03:42:05.777400 30906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59: Bootstrap starting.
I20260806 03:42:05.778604 30906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:05.778991 30906 log.cc:824] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:05.779796 30906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59: No bootstrap required, opened a new log
I20260806 03:42:05.781682 30906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b2dc5e421841ba8838339449038f59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35967 } }
I20260806 03:42:05.781879 30906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:05.781986 30906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b2dc5e421841ba8838339449038f59, State: Initialized, Role: FOLLOWER
I20260806 03:42:05.782177 30906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b2dc5e421841ba8838339449038f59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35967 } }
I20260806 03:42:05.782338 30906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:05.782389 30906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:05.782442 30906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:05.783433 30906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b2dc5e421841ba8838339449038f59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35967 } }
I20260806 03:42:05.783680 30906 leader_election.cc:302] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36b2dc5e421841ba8838339449038f59; no voters:
I20260806 03:42:05.783915 30906 leader_election.cc:288] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:05.783988 30912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:05.784142 30912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [term 1 LEADER]: Becoming Leader. State: Replica: 36b2dc5e421841ba8838339449038f59, State: Running, Role: LEADER
I20260806 03:42:05.784292 30912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b2dc5e421841ba8838339449038f59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35967 } }
I20260806 03:42:05.784430 30906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:05.784812 30912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b2dc5e421841ba8838339449038f59. Latest consensus state: current_term: 1 leader_uuid: "36b2dc5e421841ba8838339449038f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b2dc5e421841ba8838339449038f59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35967 } } }
I20260806 03:42:05.784910 30912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:05.784826 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36b2dc5e421841ba8838339449038f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b2dc5e421841ba8838339449038f59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35967 } } }
I20260806 03:42:05.785072 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:05.785229 30918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:05.786412 30918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:05.789165 30918 catalog_manager.cc:1347] Generated new cluster ID: a00446b2a544400f9c7d8f144a178858
I20260806 03:42:05.789223 30918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:05 dist-test-slave-10pc krb5kdc[30824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:05.797458 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:05.777874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.30.24.62:35967)
0806 03:42:05.778225 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:05.778249 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:05.778474 (+ 225us) server_negotiation.cc:407] Connection header received
0806 03:42:05.781666 (+ 3192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:05.781673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:05.781765 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:05.781911 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:05.783694 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:05.785402 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:05.787298 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:05.787952 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:05.788080 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:05.792591 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:05.792595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:05.792595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:05.792606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:05.794750 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:05.795255 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:05.795259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:05.795259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:05.795380 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:05.795641 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:05.795642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:05.795643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:05.795929 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:05.796080 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:05.797027 (+ 947us) server_negotiation.cc:299] Negotiation successful
0806 03:42:05.797196 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260806 03:42:05.810715 30918 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:05.811587 30918 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:05.820672 30918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36b2dc5e421841ba8838339449038f59: Generated new TSK 0
I20260806 03:42:05.820911 30918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:05.854010 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35967
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:05.952324 30933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:05.952623 30933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:05.952669 30933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:05.953855 30933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:05.953922 30933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:05.953967 30933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:05.955446 30933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:05.955811 30933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:05.956058 30933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:05.961512 30939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:05.961536 30938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:05.961642 30941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:05.961877 30933 server_base.cc:1034] running on GCE node
I20260806 03:42:05.962062 30933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:05.962268 30933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:05.963438 30933 hybrid_clock.cc:648] HybridClock initialized: now 1785987725963433 us; error 39 us; skew 500 ppm
Aug 06 03:42:05 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:05.966360 30933 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:05.966804 30933 webserver.cc:460] Webserver started at http://127.30.24.1:41621/ using document root <none> and password file <none>
I20260806 03:42:05.967034 30933 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:05.967106 30933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:05.967257 30933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:05.968336 30933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "afcdded0cb304c71bbdd7c2e4c7cc7d4"
format_stamp: "Formatted at 2026-08-06 03:42:05 on dist-test-slave-10pc"
server_key: "3b6d07b5d9f704c94b130099aee79634"
server_key_iv: "dacc03562a8430ba7ad52224177c0e2a"
server_key_version: "encryptionkey@0"
I20260806 03:42:05.968777 30933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "afcdded0cb304c71bbdd7c2e4c7cc7d4"
format_stamp: "Formatted at 2026-08-06 03:42:05 on dist-test-slave-10pc"
server_key: "3b6d07b5d9f704c94b130099aee79634"
server_key_iv: "dacc03562a8430ba7ad52224177c0e2a"
server_key_version: "encryptionkey@0"
I20260806 03:42:05.970202 30933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:05.970961 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:05.971149 30933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:05.971249 30933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "afcdded0cb304c71bbdd7c2e4c7cc7d4"
format_stamp: "Formatted at 2026-08-06 03:42:05 on dist-test-slave-10pc"
server_key: "3b6d07b5d9f704c94b130099aee79634"
server_key_iv: "dacc03562a8430ba7ad52224177c0e2a"
server_key_version: "encryptionkey@0"
I20260806 03:42:05.971334 30933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:05.993919 30933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:05.995882 30933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:05.996068 30933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:05.996426 30933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:05.996860 30933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:05.996953 30933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:05.996996 30933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:05.997056 30933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:06.005496 30933 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:42545
I20260806 03:42:06.005532 31066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:42545 every 8 connection(s)
I20260806 03:42:06.006125 30933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:06.011373 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 30933
I20260806 03:42:06.011467 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:06.011633 30816 external_mini_cluster.cc:1428] Setting key 11472d9ff3dd2ee361392ab384cdbc1e
Aug 06 03:42:06 dist-test-slave-10pc krb5kdc[30824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:06.016944 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:06.007224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:47497 (local address 127.30.24.62:35967)
0806 03:42:06.007278 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:06.007280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:06.007528 (+ 248us) server_negotiation.cc:407] Connection header received
0806 03:42:06.008033 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:06.008034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:06.008049 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:06.008132 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:06.009336 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:06.009885 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:06.010878 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:06.011123 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:06.011200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:06.013724 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:06.013726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:06.013726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:06.013734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:06.015323 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:06.015783 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:06.015784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:06.015785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:06.015852 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:06.016114 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:06.016115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:06.016115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:06.016276 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:06.016357 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:06.016668 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 03:42:06.016794 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:42:06.016927 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:06.007190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35967 (local address 127.30.24.1:47497)
0806 03:42:06.007413 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:06.007445 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:06.007938 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:06.008193 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:06.008197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:06.008997 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:06.009226 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:06.009232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:06.009976 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:06.009977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:06.010792 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:06.010794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:06.011212 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:06.011213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:06.011323 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:06.012007 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:06.012018 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:06.013628 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:06.015429 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:06.015436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:06.015437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:06.015704 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:06.015926 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:06.015936 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:06.015937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:06.016048 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:06.016345 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:06.016350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:06.016527 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 03:42:06.016616 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 03:42:06.016757 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260806 03:42:06.017514 31067 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35967
I20260806 03:42:06.017642 31067 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:06.017839 31067 heartbeater.cc:511] Master 127.30.24.62:35967 requested a full tablet report, sending...
I20260806 03:42:06.018890 30869 ts_manager.cc:194] Registered new tserver with Master: afcdded0cb304c71bbdd7c2e4c7cc7d4 (127.30.24.1:42545)
I20260806 03:42:06.020041 30869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:47497
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
W20260806 03:42:06.887099 30816 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.30.24.2: real 0.875s user 0.001s sys 0.003s
I20260806 03:42:06.887440 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35967
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:06.975615 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:06.975826 31073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:06.975852 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:06.977061 31073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:06.977103 31073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:06.977131 31073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:06.978636 31073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:06.978904 31073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:06.979152 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:06.984349 31079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:06.984406 31081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:06.984355 31078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:06.984735 31073 server_base.cc:1034] running on GCE node
I20260806 03:42:06.985070 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:06.985406 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:06.986583 31073 hybrid_clock.cc:648] HybridClock initialized: now 1785987726986569 us; error 45 us; skew 500 ppm
Aug 06 03:42:06 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987726, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:06.989272 31073 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:06.989858 31073 webserver.cc:460] Webserver started at http://127.30.24.2:33853/ using document root <none> and password file <none>
I20260806 03:42:06.990101 31073 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:06.990180 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:06.990331 31073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:06.991526 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "efce3d3d948044c18678fd5f7c3e3af2"
format_stamp: "Formatted at 2026-08-06 03:42:06 on dist-test-slave-10pc"
server_key: "f46b4b2be468ee92ae649ca17184616a"
server_key_iv: "b7ecfcdecc542ec01142b648bc521ee6"
server_key_version: "encryptionkey@0"
I20260806 03:42:06.991906 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "efce3d3d948044c18678fd5f7c3e3af2"
format_stamp: "Formatted at 2026-08-06 03:42:06 on dist-test-slave-10pc"
server_key: "f46b4b2be468ee92ae649ca17184616a"
server_key_iv: "b7ecfcdecc542ec01142b648bc521ee6"
server_key_version: "encryptionkey@0"
I20260806 03:42:06.993562 31073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:06.994531 31088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:06.994726 31073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:06.994832 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "efce3d3d948044c18678fd5f7c3e3af2"
format_stamp: "Formatted at 2026-08-06 03:42:06 on dist-test-slave-10pc"
server_key: "f46b4b2be468ee92ae649ca17184616a"
server_key_iv: "b7ecfcdecc542ec01142b648bc521ee6"
server_key_version: "encryptionkey@0"
I20260806 03:42:06.994923 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:07.021302 31067 heartbeater.cc:503] Master 127.30.24.62:35967 was elected leader, sending a full tablet report...
I20260806 03:42:07.032681 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:07.034804 31073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:07.034991 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:07.035265 31073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:07.035650 31073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:07.035707 31073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:07.035742 31073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:07.035771 31073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:07.043123 31073 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:46007
I20260806 03:42:07.043176 31205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:46007 every 8 connection(s)
I20260806 03:42:07.043565 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987726, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:07.053310 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31073
I20260806 03:42:07.053432 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:07.053607 30816 external_mini_cluster.cc:1428] Setting key de416101ce42c4b8844eb68b5bae4b40
I20260806 03:42:07.054003 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:07.044438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:57143 (local address 127.30.24.62:35967)
0806 03:42:07.044602 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:07.044605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:07.044828 (+ 223us) server_negotiation.cc:407] Connection header received
0806 03:42:07.045354 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:07.045355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:07.045367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:07.045431 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:07.046818 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.047337 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.048333 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.048549 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.048601 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:07.050812 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:07.050813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:07.050814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:07.050821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:07.052416 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.052856 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.052856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.052857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.052916 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.053191 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.053191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.053191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.053312 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:07.053365 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:07.053733 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 03:42:07.053854 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260806 03:42:07.054188 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:07.044398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35967 (local address 127.30.24.2:57143)
0806 03:42:07.044689 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:07.044731 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:07.045255 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:07.045486 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:07.045490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:07.046122 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:07.046716 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:07.046722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.047421 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.047422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:07.048254 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:07.048256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.048620 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.048621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:07.048730 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:07.049222 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:07.049231 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:07.050693 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:07.052552 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:07.052559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:07.052561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:07.052788 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:07.052981 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:07.052984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:07.052984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:07.053122 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:07.053376 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:07.053384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:07.053570 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 03:42:07.053693 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 03:42:07.053879 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":214,"threads_started":1}
I20260806 03:42:07.054653 31206 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35967
I20260806 03:42:07.054765 31206 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:07.054996 31206 heartbeater.cc:511] Master 127.30.24.62:35967 requested a full tablet report, sending...
I20260806 03:42:07.055460 30869 ts_manager.cc:194] Registered new tserver with Master: efce3d3d948044c18678fd5f7c3e3af2 (127.30.24.2:46007)
I20260806 03:42:07.055917 30869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:57143
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
W20260806 03:42:07.676844 30816 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.30.24.3: real 0.622s user 0.000s sys 0.004s
I20260806 03:42:07.677234 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35967
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:07.761380 31214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:07.761628 31214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:07.761669 31214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:07.762871 31214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:07.762943 31214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:07.762989 31214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:07.764504 31214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:07.764789 31214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:07.765045 31214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:07.770346 31222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:07.770354 31220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:07.770354 31219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:07.770701 31214 server_base.cc:1034] running on GCE node
I20260806 03:42:07.770911 31214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:07.771180 31214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:07.772369 31214 hybrid_clock.cc:648] HybridClock initialized: now 1785987727772346 us; error 37 us; skew 500 ppm
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:07.775130 31214 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:07.775635 31214 webserver.cc:460] Webserver started at http://127.30.24.3:44449/ using document root <none> and password file <none>
I20260806 03:42:07.775839 31214 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:07.775882 31214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:07.775975 31214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:07.777097 31214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "36bef89873e34c66ace2a7214b183a3c"
format_stamp: "Formatted at 2026-08-06 03:42:07 on dist-test-slave-10pc"
server_key: "fe6b8bd9cae45838172df9f02d39a303"
server_key_iv: "6e4d259de71da016dc540ca1fab35d5b"
server_key_version: "encryptionkey@0"
I20260806 03:42:07.777524 31214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "36bef89873e34c66ace2a7214b183a3c"
format_stamp: "Formatted at 2026-08-06 03:42:07 on dist-test-slave-10pc"
server_key: "fe6b8bd9cae45838172df9f02d39a303"
server_key_iv: "6e4d259de71da016dc540ca1fab35d5b"
server_key_version: "encryptionkey@0"
I20260806 03:42:07.778846 31214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:07.779547 31230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:07.779752 31214 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:07.779838 31214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "36bef89873e34c66ace2a7214b183a3c"
format_stamp: "Formatted at 2026-08-06 03:42:07 on dist-test-slave-10pc"
server_key: "fe6b8bd9cae45838172df9f02d39a303"
server_key_iv: "6e4d259de71da016dc540ca1fab35d5b"
server_key_version: "encryptionkey@0"
I20260806 03:42:07.779896 31214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:07.792201 31214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:07.794261 31214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:07.794376 31214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:07.794566 31214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:07.794924 31214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:07.794966 31214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:07.794988 31214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:07.795001 31214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:07.801468 31214 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:35385
I20260806 03:42:07.801545 31349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:35385 every 8 connection(s)
I20260806 03:42:07.801898 31214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:07.803020 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31214
I20260806 03:42:07.803107 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:07.803309 30816 external_mini_cluster.cc:1428] Setting key d441a1f3e0ce72123d07d3da07138929
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:07.812067 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:07.802692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:58837 (local address 127.30.24.62:35967)
0806 03:42:07.802827 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:07.802829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:07.802948 (+ 119us) server_negotiation.cc:407] Connection header received
0806 03:42:07.803593 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:07.803594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:07.803612 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:07.803695 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:07.805106 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.805628 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.806606 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.806802 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.806859 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:07.809444 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:07.809445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:07.809446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:07.809452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:07.810771 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.811224 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.811224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.811225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.811278 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.811508 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.811509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.811509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.811617 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:07.811668 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:07.811928 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 03:42:07.811996 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260806 03:42:07.812378 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:07.802627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35967 (local address 127.30.24.3:58837)
0806 03:42:07.802802 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:07.802845 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:07.803477 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:07.803826 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:07.803831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:07.804739 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:07.804957 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:07.804963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.805717 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.805718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:07.806520 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:07.806522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.806905 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.806906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:07.807039 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:07.807716 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:07.807725 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:07.809353 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:07.810856 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:07.810863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:07.810865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:07.811154 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:07.811344 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:07.811344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:07.811344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:07.811453 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:07.811680 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:07.811685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:07.811812 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 03:42:07.811904 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 03:42:07.812092 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260806 03:42:07.812960 31350 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35967
I20260806 03:42:07.813076 31350 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:07.813233 31350 heartbeater.cc:511] Master 127.30.24.62:35967 requested a full tablet report, sending...
I20260806 03:42:07.813645 30869 ts_manager.cc:194] Registered new tserver with Master: 36bef89873e34c66ace2a7214b183a3c (127.30.24.3:35385)
I20260806 03:42:07.814011 30869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:58837
I20260806 03:42:07.815398 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:07.839540 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:07.831932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.30.24.1:42545)
0806 03:42:07.832089 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:07.832095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:07.832108 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:07.832195 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:07.832199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:07.832212 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:07.832283 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:07.832907 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.833611 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.834327 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.834635 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.834690 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:07.836938 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:07.836947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:07.836947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:07.836955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:07.838424 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.838775 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.838779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.838779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.838856 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.839058 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.839059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.839059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.839219 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:07.839351 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:07.839395 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 03:42:07.839461 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260806 03:42:07.840109 30979 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39136
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[30824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:07.864166 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:07.857269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.30.24.1:42545)
0806 03:42:07.857366 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:07.857369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:07.857384 (+ 15us) server_negotiation.cc:407] Connection header received
0806 03:42:07.857471 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:07.857472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:07.857491 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:07.857563 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:07.858140 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.858626 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.859318 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:07.859508 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:07.859559 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:07.861800 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:07.861801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:07.861802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:07.861808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:07.863213 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.863522 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.863523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.863523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.863577 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:07.863752 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:07.863753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:07.863753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:07.863842 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:07.863939 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:07.863999 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 03:42:07.864060 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 03:42:07.864784 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 30933
I20260806 03:42:07.871325 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31073
I20260806 03:42:07.876659 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31214
I20260806 03:42:07.881928 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 30837
2026-08-06T03:42:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3458 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[31366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[31366](info): set up 2 sockets
Aug 06 03:42:07 dist-test-slave-10pc krb5kdc[31366](info): commencing operation
krb5kdc: starting...
W20260806 03:42:08.944919 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:09.067267 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35963
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:35963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:09.150873 31379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:09.151135 31379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:09.151176 31379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:09.152334 31379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:09.152406 31379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:09.152431 31379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:09.152455 31379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:09.152516 31379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:09.154032 31379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:35963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:09.154317 31379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:09.154559 31379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:09.159673 31384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.159667 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.159688 31385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:09.159737 31379 server_base.cc:1034] running on GCE node
I20260806 03:42:09.160084 31379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:09.160405 31379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:09.161600 31379 hybrid_clock.cc:648] HybridClock initialized: now 1785987729161576 us; error 41 us; skew 500 ppm
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:09.164027 31379 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:09.164572 31379 webserver.cc:460] Webserver started at http://127.30.24.62:32927/ using document root <none> and password file <none>
I20260806 03:42:09.164803 31379 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:09.164872 31379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:09.165019 31379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:09.166093 31379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "d4de0e0201ad429e93c5674523d8dd59"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "86368feb4b009a0e9f66009125749860"
server_key_iv: "a7077b241e87d57cdf5d2a02a4ae2cfe"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.166502 31379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "d4de0e0201ad429e93c5674523d8dd59"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "86368feb4b009a0e9f66009125749860"
server_key_iv: "a7077b241e87d57cdf5d2a02a4ae2cfe"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.167959 31379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:09.168820 31395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.168994 31379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.169107 31379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "d4de0e0201ad429e93c5674523d8dd59"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "86368feb4b009a0e9f66009125749860"
server_key_iv: "a7077b241e87d57cdf5d2a02a4ae2cfe"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.169203 31379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:09.182082 31379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:09.183976 31379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:09.184161 31379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:09.188366 31379 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:35963
I20260806 03:42:09.188401 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:35963 every 8 connection(s)
I20260806 03:42:09.188813 31379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:09.189715 31450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:09.192083 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59: Bootstrap starting.
I20260806 03:42:09.192818 31450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:09.192935 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31379
I20260806 03:42:09.193042 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:09.193181 31450 log.cc:824] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:09.193214 30816 external_mini_cluster.cc:1428] Setting key ac1ca5c1612ab024b54c2abb0f5eb24a
I20260806 03:42:09.194226 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59: No bootstrap required, opened a new log
I20260806 03:42:09.196867 31450 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4de0e0201ad429e93c5674523d8dd59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35963 } }
I20260806 03:42:09.197046 31450 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:09.197080 31450 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de0e0201ad429e93c5674523d8dd59, State: Initialized, Role: FOLLOWER
I20260806 03:42:09.197196 31450 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4de0e0201ad429e93c5674523d8dd59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35963 } }
I20260806 03:42:09.197273 31450 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:09.197304 31450 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:09.197351 31450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:09.197903 31450 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4de0e0201ad429e93c5674523d8dd59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35963 } }
I20260806 03:42:09.198037 31450 leader_election.cc:302] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4de0e0201ad429e93c5674523d8dd59; no voters:
I20260806 03:42:09.198235 31450 leader_election.cc:288] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:09.198307 31455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:09.198490 31455 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [term 1 LEADER]: Becoming Leader. State: Replica: d4de0e0201ad429e93c5674523d8dd59, State: Running, Role: LEADER
I20260806 03:42:09.198544 31450 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:09.198704 31455 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4de0e0201ad429e93c5674523d8dd59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35963 } }
I20260806 03:42:09.199110 31457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4de0e0201ad429e93c5674523d8dd59. Latest consensus state: current_term: 1 leader_uuid: "d4de0e0201ad429e93c5674523d8dd59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4de0e0201ad429e93c5674523d8dd59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35963 } } }
I20260806 03:42:09.199260 31457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:09.199546 31464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:09.200045 31456 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4de0e0201ad429e93c5674523d8dd59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4de0e0201ad429e93c5674523d8dd59" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35963 } } }
I20260806 03:42:09.200098 31464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:09.200222 31456 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:09.202008 31464 catalog_manager.cc:1347] Generated new cluster ID: 165866db9bc74d6cacf150051add9d06
I20260806 03:42:09.202057 31464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:09.207201 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.193794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.30.24.62:35963)
0806 03:42:09.193972 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.193980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.194008 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:42:09.194533 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.194540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.194589 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.194726 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.195495 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.196367 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.197477 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.197895 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.197982 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.201059 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.201064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.201064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.201076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.205434 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.205887 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.205890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.205891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.205998 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.206250 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.206251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.206251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.206542 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.206696 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.206856 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.206992 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260806 03:42:09.215618 31464 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:09.216162 31464 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:09.219666 31464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4de0e0201ad429e93c5674523d8dd59: Generated new TSK 0
I20260806 03:42:09.219849 31464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:09.255024 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35963
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:09.340260 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:09.340497 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:09.340524 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:09.341734 31476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:09.341779 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:09.341836 31476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:09.343365 31476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35963
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:09.343631 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:09.343879 31476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:09.349161 31481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.349162 31482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.349162 31484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:09.349364 31476 server_base.cc:1034] running on GCE node
I20260806 03:42:09.349685 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:09.349908 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:09.351058 31476 hybrid_clock.cc:648] HybridClock initialized: now 1785987729351036 us; error 36 us; skew 500 ppm
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:09.353915 31476 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:09.354331 31476 webserver.cc:460] Webserver started at http://127.30.24.1:45573/ using document root <none> and password file <none>
I20260806 03:42:09.354542 31476 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:09.354581 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:09.354676 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:09.355707 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "bfb777754dee41f58b81301de41c79e2"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "6e23b916d089ebad198b883b4c231832"
server_key_iv: "ba6690f57c66d5f108eaab7a910ffef6"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.356135 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "bfb777754dee41f58b81301de41c79e2"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "6e23b916d089ebad198b883b4c231832"
server_key_iv: "ba6690f57c66d5f108eaab7a910ffef6"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.357560 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:09.358310 31491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.358507 31476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:09.358608 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "bfb777754dee41f58b81301de41c79e2"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "6e23b916d089ebad198b883b4c231832"
server_key_iv: "ba6690f57c66d5f108eaab7a910ffef6"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.358685 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:09.377058 31476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:09.379685 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:09.379866 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:09.380152 31476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:09.380574 31476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:09.380633 31476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.380671 31476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:09.380698 31476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.386812 31476 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:44853
I20260806 03:42:09.386857 31608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:44853 every 8 connection(s)
I20260806 03:42:09.387213 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:09.390758 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31476
I20260806 03:42:09.390868 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:09.391033 30816 external_mini_cluster.cc:1428] Setting key 4409933cfaa3c18733a1a21166093218
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:09.397797 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.388013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:39835 (local address 127.30.24.62:35963)
0806 03:42:09.388061 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.388063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.388321 (+ 258us) server_negotiation.cc:407] Connection header received
0806 03:42:09.389481 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.389482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.389495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.389534 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.390204 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.390726 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.392057 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.392273 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.392348 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.394851 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.394852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.394853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.394860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.396267 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.396777 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.396778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.396778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.396842 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.397113 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.397114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.397115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.397234 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.397309 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.397605 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.397705 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:09.397954 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.387999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35963 (local address 127.30.24.1:39835)
0806 03:42:09.388188 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.388227 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.389361 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.389614 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.389618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.389902 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:09.390100 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.390104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.390836 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.390837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.391952 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.391955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.392360 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.392361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.392533 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.393104 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:09.393113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:09.394765 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:09.396386 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:09.396394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:09.396396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:09.396695 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:09.396912 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:09.396913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:09.396913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:09.397032 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:09.397314 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:09.397322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:09.397464 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.397562 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.397720 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260806 03:42:09.398468 31609 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35963
I20260806 03:42:09.398586 31609 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:09.398798 31609 heartbeater.cc:511] Master 127.30.24.62:35963 requested a full tablet report, sending...
I20260806 03:42:09.399381 31413 ts_manager.cc:194] Registered new tserver with Master: bfb777754dee41f58b81301de41c79e2 (127.30.24.1:44853)
I20260806 03:42:09.400130 31413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:39835
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:09.419449 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35963
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:09.504176 31614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:09.504470 31614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:09.504498 31614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:09.505671 31614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:09.505714 31614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:09.505743 31614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:09.507249 31614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35963
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:09.507457 31614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:09.507649 31614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:09.512725 31620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.512800 31622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.512763 31619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:09.512959 31614 server_base.cc:1034] running on GCE node
I20260806 03:42:09.513175 31614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:09.513437 31614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:09.514597 31614 hybrid_clock.cc:648] HybridClock initialized: now 1785987729514574 us; error 54 us; skew 500 ppm
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:09.520985 31614 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:09.521468 31614 webserver.cc:460] Webserver started at http://127.30.24.2:33565/ using document root <none> and password file <none>
I20260806 03:42:09.521740 31614 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:09.521813 31614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:09.521971 31614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:09.523082 31614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "8178428e0b6644e9949f06f13b654fa4"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "9d7a3965bb4ab729a423d158dac056d3"
server_key_iv: "e90475ab90e4354314674d87b7bc6413"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.523464 31614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "8178428e0b6644e9949f06f13b654fa4"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "9d7a3965bb4ab729a423d158dac056d3"
server_key_iv: "e90475ab90e4354314674d87b7bc6413"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.524919 31614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:09.525704 31629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.525943 31614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:09.526070 31614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "8178428e0b6644e9949f06f13b654fa4"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "9d7a3965bb4ab729a423d158dac056d3"
server_key_iv: "e90475ab90e4354314674d87b7bc6413"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.526155 31614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:09.547261 31614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:09.549290 31614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:09.549469 31614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:09.549731 31614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:09.550105 31614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:09.550184 31614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.550251 31614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:09.550282 31614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.556475 31614 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:43371
I20260806 03:42:09.556566 31747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:43371 every 8 connection(s)
I20260806 03:42:09.556911 31614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:09.565258 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31614
I20260806 03:42:09.565371 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:09.565537 30816 external_mini_cluster.cc:1428] Setting key b750134f91609d038e09fb72f0ea7cf9
I20260806 03:42:09.567639 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.557675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35963 (local address 127.30.24.2:43725)
0806 03:42:09.557833 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.557871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.558883 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.559160 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.559164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.559427 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:09.559636 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.559639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.560361 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.560363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.561340 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.561343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.561783 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.561784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.561908 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.562496 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:09.562504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:09.564075 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:09.566141 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:09.566147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:09.566149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:09.566454 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:09.566705 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:09.566705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:09.566706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:09.566810 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:09.567108 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:09.567131 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:09.567257 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.567342 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.567466 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":101,"threads_started":1}
I20260806 03:42:09.567637 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.557713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:43725 (local address 127.30.24.62:35963)
0806 03:42:09.557762 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.557765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.557960 (+ 195us) server_negotiation.cc:407] Connection header received
0806 03:42:09.559007 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.559008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.559021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.559065 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.559739 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.560254 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.561465 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.561683 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.561759 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.564180 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.564182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.564182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.564190 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.565844 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.566559 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.566561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.566561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.566630 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.566880 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.566881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.566882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.567039 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.567117 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.567394 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.567490 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:09.568163 31748 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35963
I20260806 03:42:09.568293 31748 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:09.568537 31748 heartbeater.cc:511] Master 127.30.24.62:35963 requested a full tablet report, sending...
I20260806 03:42:09.568953 31413 ts_manager.cc:194] Registered new tserver with Master: 8178428e0b6644e9949f06f13b654fa4 (127.30.24.2:43371)
I20260806 03:42:09.569337 31413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:43725
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:09.591517 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35963
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:09.675920 31753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:09.676123 31753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:09.676149 31753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:09.677387 31753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:09.677433 31753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:09.677459 31753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:09.678949 31753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35963
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:09.679200 31753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:09.679455 31753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:09.684682 31753 server_base.cc:1034] running on GCE node
W20260806 03:42:09.684654 31759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.684664 31760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:09.684664 31762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:09.685144 31753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:09.685412 31753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:09.686563 31753 hybrid_clock.cc:648] HybridClock initialized: now 1785987729686558 us; error 36 us; skew 500 ppm
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:09.689183 31753 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:09.689628 31753 webserver.cc:460] Webserver started at http://127.30.24.3:36355/ using document root <none> and password file <none>
I20260806 03:42:09.689858 31753 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:09.689924 31753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:09.690075 31753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:09.691134 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "446a6f1ee7b349a8a966125b8c062238"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "91f8741959ae58a26b8d93678903fcbc"
server_key_iv: "c4482b4e3ea92db4a104b3fe5b84b037"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.691511 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "446a6f1ee7b349a8a966125b8c062238"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "91f8741959ae58a26b8d93678903fcbc"
server_key_iv: "c4482b4e3ea92db4a104b3fe5b84b037"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.692962 31753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:09.693670 31769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.693871 31753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:09.693970 31753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "446a6f1ee7b349a8a966125b8c062238"
format_stamp: "Formatted at 2026-08-06 03:42:09 on dist-test-slave-10pc"
server_key: "91f8741959ae58a26b8d93678903fcbc"
server_key_iv: "c4482b4e3ea92db4a104b3fe5b84b037"
server_key_version: "encryptionkey@0"
I20260806 03:42:09.694043 31753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:09.706246 31753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:09.708632 31753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:09.708793 31753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:09.709021 31753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:09.709422 31753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:09.709481 31753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.709525 31753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:09.709553 31753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:09.715891 31753 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:33647
I20260806 03:42:09.715932 31887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:33647 every 8 connection(s)
I20260806 03:42:09.716339 31753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:09.725878 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.717102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:60523 (local address 127.30.24.62:35963)
0806 03:42:09.717152 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.717155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.717354 (+ 199us) server_negotiation.cc:407] Connection header received
0806 03:42:09.718391 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.718393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.718407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.718464 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.719127 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.719754 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.720896 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.721101 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.721178 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.723410 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.723412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.723412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.723419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.724689 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.725070 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.725071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.725071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.725122 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.725340 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.725340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.725340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.725443 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.725495 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.725744 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.725809 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:09.726084 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.717054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35963 (local address 127.30.24.3:60523)
0806 03:42:09.717202 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.717254 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.718263 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.718547 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.718551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.718819 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:09.719014 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.719019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.719848 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.719850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.720811 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.720814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.721206 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.721207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.721309 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.721933 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:09.721942 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:09.723323 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:09.724765 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:09.724772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:09.724774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:09.724986 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:09.725182 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:09.725183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:09.725184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:09.725283 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:09.725500 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:09.725504 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:09.725625 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.725709 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.725830 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260806 03:42:09.726431 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31753
I20260806 03:42:09.726526 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:09.726657 31888 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35963
I20260806 03:42:09.726696 30816 external_mini_cluster.cc:1428] Setting key bbd25e337384728841a7b94da329d696
I20260806 03:42:09.726783 31888 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:09.726966 31888 heartbeater.cc:511] Master 127.30.24.62:35963 requested a full tablet report, sending...
I20260806 03:42:09.727394 31413 ts_manager.cc:194] Registered new tserver with Master: 446a6f1ee7b349a8a966125b8c062238 (127.30.24.3:33647)
I20260806 03:42:09.727797 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:42:09.727842 31413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:60523
I20260806 03:42:09.735276 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.729784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.30.24.62:35963)
0806 03:42:09.729826 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.729829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.729931 (+ 102us) server_negotiation.cc:407] Connection header received
0806 03:42:09.729986 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.729987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.729995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.730037 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.730603 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.731083 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.731768 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.731939 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.731987 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.733161 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.733162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.733162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.733167 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.734309 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.734618 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.734619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.734619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.734668 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.734832 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.734832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.734833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.734964 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.735041 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.735126 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.735185 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:42:09.737970 31413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:09.739773 31413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:09.747097 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.742797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44853 (local address 127.0.0.1:52806)
0806 03:42:09.742992 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.743047 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.743114 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.743357 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.743361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.743364 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:09.743441 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.743443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.744877 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.744878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.745832 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.745835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.746858 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.746859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.746920 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.746932 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.746991 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.747037 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260806 03:42:09.747504 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.743012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:43371 (local address 127.0.0.1:33254)
0806 03:42:09.743488 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.743498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.743558 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.743740 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.743741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.743743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:09.743784 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.743786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.745017 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.745017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.746270 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.746272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.747260 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.747261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.747331 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.747335 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.747396 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.747438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":58,"thread_start_us":66,"threads_started":1}
I20260806 03:42:09.747131 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.742907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.30.24.1:44853)
0806 03:42:09.743119 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.743125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.743138 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:09.743213 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.743215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.743234 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.743333 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:09.743542 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.744749 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.746037 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.746770 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.746849 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.746909 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.747004 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.747043 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260806 03:42:09.747583 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.743257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.30.24.2:43371)
0806 03:42:09.743484 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.743489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.743553 (+ 64us) server_negotiation.cc:407] Connection header received
0806 03:42:09.743629 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.743631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.743648 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.743731 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:09.743849 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.744898 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.746356 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.747090 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.747164 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.747344 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.747438 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.747480 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260806 03:42:09.748276 31680 tablet_service.cc:1467] Processing CreateTablet for tablet 44fefd007fa74f7bacce15f5e637b4b4 (DEFAULT_TABLE table=test-table [id=2f592cc5f4ce45b3ba8b5995dbe797ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:09.748606 31680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44fefd007fa74f7bacce15f5e637b4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:09.748723 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.742799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:33647 (local address 127.0.0.1:38758)
0806 03:42:09.743232 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.743243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.743293 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.743613 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.743614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.743617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:09.743671 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.743673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.744949 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.744950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.746268 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.746272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.748371 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.748372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.748542 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.748547 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.748583 (+ 36us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.748629 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":44,"thread_start_us":83,"threads_started":1}
I20260806 03:42:09.749121 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.742884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.30.24.3:33647)
0806 03:42:09.743437 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.743442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.743452 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:09.743492 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.743494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.743513 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.743781 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:09.743865 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.744843 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.747383 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.748260 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.748922 (+ 662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.748995 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.749016 (+ 21us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.749056 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":144,"threads_started":1}
I20260806 03:42:09.749687 31821 tablet_service.cc:1467] Processing CreateTablet for tablet 44fefd007fa74f7bacce15f5e637b4b4 (DEFAULT_TABLE table=test-table [id=2f592cc5f4ce45b3ba8b5995dbe797ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:09.750013 31821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44fefd007fa74f7bacce15f5e637b4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:09.750411 31542 tablet_service.cc:1467] Processing CreateTablet for tablet 44fefd007fa74f7bacce15f5e637b4b4 (DEFAULT_TABLE table=test-table [id=2f592cc5f4ce45b3ba8b5995dbe797ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:09.750732 31542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44fefd007fa74f7bacce15f5e637b4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:09.751688 31906 tablet_bootstrap.cc:492] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: Bootstrap starting.
I20260806 03:42:09.752504 31906 tablet_bootstrap.cc:654] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:09.753285 31906 log.cc:824] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:09.753427 31907 tablet_bootstrap.cc:492] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238: Bootstrap starting.
I20260806 03:42:09.754181 31907 tablet_bootstrap.cc:654] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:09.754249 31906 tablet_bootstrap.cc:492] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: No bootstrap required, opened a new log
I20260806 03:42:09.754348 31906 ts_tablet_manager.cc:1397] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:42:09.754709 31907 log.cc:824] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:09.755414 31907 tablet_bootstrap.cc:492] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238: No bootstrap required, opened a new log
I20260806 03:42:09.755499 31907 ts_tablet_manager.cc:1397] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:09.756001 31906 raft_consensus.cc:348] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.756202 31906 raft_consensus.cc:374] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:09.756258 31906 raft_consensus.cc:729] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8178428e0b6644e9949f06f13b654fa4, State: Initialized, Role: FOLLOWER
I20260806 03:42:09.756393 31906 consensus_queue.cc:260] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.756654 31909 tablet_bootstrap.cc:492] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2: Bootstrap starting.
I20260806 03:42:09.756716 31906 ts_tablet_manager.cc:1428] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 03:42:09.756942 31748 heartbeater.cc:503] Master 127.30.24.62:35963 was elected leader, sending a full tablet report...
I20260806 03:42:09.757310 31909 tablet_bootstrap.cc:654] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:09.757602 31907 raft_consensus.cc:348] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.757800 31907 raft_consensus.cc:374] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:09.757849 31907 raft_consensus.cc:729] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446a6f1ee7b349a8a966125b8c062238, State: Initialized, Role: FOLLOWER
I20260806 03:42:09.757881 31909 log.cc:824] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:09.757993 31907 consensus_queue.cc:260] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.758308 31888 heartbeater.cc:503] Master 127.30.24.62:35963 was elected leader, sending a full tablet report...
I20260806 03:42:09.758291 31907 ts_tablet_manager.cc:1428] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:09.759191 31909 tablet_bootstrap.cc:492] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2: No bootstrap required, opened a new log
I20260806 03:42:09.759287 31909 ts_tablet_manager.cc:1397] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:42:09.760968 31909 raft_consensus.cc:348] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.761147 31909 raft_consensus.cc:374] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:09.761193 31909 raft_consensus.cc:729] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb777754dee41f58b81301de41c79e2, State: Initialized, Role: FOLLOWER
I20260806 03:42:09.761302 31909 consensus_queue.cc:260] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.761626 31909 ts_tablet_manager.cc:1428] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 03:42:09.761632 31609 heartbeater.cc:503] Master 127.30.24.62:35963 was elected leader, sending a full tablet report...
I20260806 03:42:09.775256 31913 raft_consensus.cc:482] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:09.775352 31913 raft_consensus.cc:504] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.775825 31913 leader_election.cc:288] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfb777754dee41f58b81301de41c79e2 (127.30.24.1:44853), 8178428e0b6644e9949f06f13b654fa4 (127.30.24.2:43371)
I20260806 03:42:09.778729 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.776001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44853 (local address 127.30.24.3:49015)
0806 03:42:09.776112 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.776127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.776183 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.776425 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.776427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.776429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:09.776521 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.776523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.777169 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.777170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.777921 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.777923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.778462 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.778463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.778548 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.778551 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.778618 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.778659 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 03:42:09.778728 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.776093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:49015 (local address 127.30.24.1:44853)
0806 03:42:09.776155 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.776158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.776171 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:09.776311 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.776312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.776322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.776410 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:09.776598 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.777077 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.778018 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.778378 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.778463 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.778474 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.778625 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.778662 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:42:09.778825 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.776199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:37481 (local address 127.30.24.2:43371)
0806 03:42:09.776263 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.776266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.776353 (+ 87us) server_negotiation.cc:407] Connection header received
0806 03:42:09.776414 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.776416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.776428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.776537 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:09.776704 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.777166 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.778105 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.778462 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.778551 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.778560 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.778728 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.778763 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:42:09.778842 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.776113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:43371 (local address 127.30.24.3:37481)
0806 03:42:09.776278 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:09.776287 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:09.776361 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:09.776546 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:09.776547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:09.776548 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:09.776625 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.776628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.777248 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.777249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.778012 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:09.778014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.778553 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.778554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:09.778639 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.778643 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:09.778719 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 03:42:09.778757 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260806 03:42:09.779155 31701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fefd007fa74f7bacce15f5e637b4b4" candidate_uuid: "446a6f1ee7b349a8a966125b8c062238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8178428e0b6644e9949f06f13b654fa4" is_pre_election: true
I20260806 03:42:09.779165 31563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fefd007fa74f7bacce15f5e637b4b4" candidate_uuid: "446a6f1ee7b349a8a966125b8c062238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb777754dee41f58b81301de41c79e2" is_pre_election: true
I20260806 03:42:09.779310 31563 raft_consensus.cc:2454] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446a6f1ee7b349a8a966125b8c062238 in term 0.
I20260806 03:42:09.779337 31701 raft_consensus.cc:2454] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446a6f1ee7b349a8a966125b8c062238 in term 0.
I20260806 03:42:09.779655 31770 leader_election.cc:302] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446a6f1ee7b349a8a966125b8c062238, 8178428e0b6644e9949f06f13b654fa4; no voters:
I20260806 03:42:09.779817 31913 raft_consensus.cc:2781] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:09.779896 31913 raft_consensus.cc:482] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:09.779929 31913 raft_consensus.cc:3037] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:09.780750 31913 raft_consensus.cc:504] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.780892 31913 leader_election.cc:288] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [CANDIDATE]: Term 1 election: Requested vote from peers bfb777754dee41f58b81301de41c79e2 (127.30.24.1:44853), 8178428e0b6644e9949f06f13b654fa4 (127.30.24.2:43371)
I20260806 03:42:09.781141 31701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fefd007fa74f7bacce15f5e637b4b4" candidate_uuid: "446a6f1ee7b349a8a966125b8c062238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8178428e0b6644e9949f06f13b654fa4"
I20260806 03:42:09.781236 31701 raft_consensus.cc:3037] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:09.781597 31563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fefd007fa74f7bacce15f5e637b4b4" candidate_uuid: "446a6f1ee7b349a8a966125b8c062238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb777754dee41f58b81301de41c79e2"
I20260806 03:42:09.781682 31563 raft_consensus.cc:3037] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:09.782246 31701 raft_consensus.cc:2454] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446a6f1ee7b349a8a966125b8c062238 in term 1.
I20260806 03:42:09.782428 31770 leader_election.cc:302] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446a6f1ee7b349a8a966125b8c062238, 8178428e0b6644e9949f06f13b654fa4; no voters:
I20260806 03:42:09.782579 31913 raft_consensus.cc:2781] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:09.782686 31563 raft_consensus.cc:2454] T 44fefd007fa74f7bacce15f5e637b4b4 P bfb777754dee41f58b81301de41c79e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446a6f1ee7b349a8a966125b8c062238 in term 1.
I20260806 03:42:09.782763 31913 raft_consensus.cc:686] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [term 1 LEADER]: Becoming Leader. State: Replica: 446a6f1ee7b349a8a966125b8c062238, State: Running, Role: LEADER
I20260806 03:42:09.782883 31913 consensus_queue.cc:237] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } }
I20260806 03:42:09.783988 31413 catalog_manager.cc:5495] T 44fefd007fa74f7bacce15f5e637b4b4 P 446a6f1ee7b349a8a966125b8c062238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 446a6f1ee7b349a8a966125b8c062238 (127.30.24.3). New cstate: current_term: 1 leader_uuid: "446a6f1ee7b349a8a966125b8c062238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb777754dee41f58b81301de41c79e2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8178428e0b6644e9949f06f13b654fa4" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446a6f1ee7b349a8a966125b8c062238" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 33647 } health_report { overall_health: HEALTHY } } }
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:09.798406 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.790683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.30.24.1:44853)
0806 03:42:09.790763 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.790766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.790836 (+ 70us) server_negotiation.cc:407] Connection header received
0806 03:42:09.790916 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.790917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.790927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.790984 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.791555 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.792022 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.792831 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.793058 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.793136 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.795526 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.795537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.795537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.795543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.797215 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.797620 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.797623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.797623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.797679 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.797852 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.797852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.797852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.798042 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.798157 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.798273 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.798339 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 03:42:09.807540 31749 tablet.cc:2369] T 44fefd007fa74f7bacce15f5e637b4b4 P 8178428e0b6644e9949f06f13b654fa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:09.823643 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.816535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.30.24.1:44853)
0806 03:42:09.816593 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.816596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.816724 (+ 128us) server_negotiation.cc:407] Connection header received
0806 03:42:09.816777 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.816778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.816788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.816839 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.817436 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.817934 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.818695 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.818911 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.818969 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.821332 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.821333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.821334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.821340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.822531 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.822872 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.822873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.822873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.822922 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.823107 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.823108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.823108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.823195 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.823266 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.823455 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.823520 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:09.847199 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:09.840418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.30.24.1:44853)
0806 03:42:09.840526 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:09.840528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:09.840597 (+ 69us) server_negotiation.cc:407] Connection header received
0806 03:42:09.840646 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:09.840648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:09.840658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:09.840716 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:09.841242 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.841699 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.842400 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:09.842571 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:09.842621 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:09.844812 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:09.844814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:09.844814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:09.844821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:09.846269 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.846563 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.846563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.846564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.846617 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:09.846769 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:09.846769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:09.846769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:09.846851 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:09.846915 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:09.847011 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 03:42:09.847068 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 03:42:09.848030 31521 tablet_service.cc:2908] Rejecting scan request for tablet 44fefd007fa74f7bacce15f5e637b4b4: Uninitialized: clean time has not yet been initialized
I20260806 03:42:09.849025 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31476
I20260806 03:42:09.856024 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31614
I20260806 03:42:09.861603 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31753
I20260806 03:42:09.867077 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31379
2026-08-06T03:42:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1985 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31930](info): set up 2 sockets
Aug 06 03:42:09 dist-test-slave-10pc krb5kdc[31930](info): commencing operation
krb5kdc: starting...
W20260806 03:42:10.925563 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:11.052336 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35721
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:35721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:11.137818 31943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:11.138031 31943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:11.138051 31943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:11.139214 31943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:11.139257 31943 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 03:42:11.139271 31943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:11.139284 31943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:11.139343 31943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:11.139360 31943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:11.140870 31943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:35721
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:11.141129 31943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:11.141399 31943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:11.146461 31951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.146451 31949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.146438 31953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:11.146809 31943 server_base.cc:1034] running on GCE node
I20260806 03:42:11.147068 31943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:11.147356 31943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:11.148525 31943 hybrid_clock.cc:648] HybridClock initialized: now 1785987731148512 us; error 38 us; skew 500 ppm
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:11.151077 31943 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:11.151521 31943 webserver.cc:460] Webserver started at http://127.30.24.62:42387/ using document root <none> and password file <none>
I20260806 03:42:11.151705 31943 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:11.151751 31943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:11.151844 31943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:11.153023 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "727601e60f4d4738882153eac52205fd"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "1cde30f752b346559fe85d7f570ca4cc"
server_key_iv: "728e592fca3a1a3d02ce4d2b73fd4df6"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.153342 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "727601e60f4d4738882153eac52205fd"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "1cde30f752b346559fe85d7f570ca4cc"
server_key_iv: "728e592fca3a1a3d02ce4d2b73fd4df6"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.154589 31943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:11.155241 31960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.155435 31943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.155527 31943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "727601e60f4d4738882153eac52205fd"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "1cde30f752b346559fe85d7f570ca4cc"
server_key_iv: "728e592fca3a1a3d02ce4d2b73fd4df6"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.155584 31943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:11.162685 31943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:11.164985 31943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:11.165119 31943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:11.169866 31943 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:35721
I20260806 03:42:11.169963 32024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:35721 every 8 connection(s)
I20260806 03:42:11.170297 31943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:11.170984 32025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.171515 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 31943
I20260806 03:42:11.171593 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:11.171782 30816 external_mini_cluster.cc:1428] Setting key 36f41add78996c7fb5c277557d268ee6
I20260806 03:42:11.173658 32025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: Bootstrap starting.
I20260806 03:42:11.174300 32025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.174604 32025 log.cc:824] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:11.175312 32025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: No bootstrap required, opened a new log
I20260806 03:42:11.177256 32025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727601e60f4d4738882153eac52205fd" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35721 } }
I20260806 03:42:11.177423 32025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.177485 32025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727601e60f4d4738882153eac52205fd, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.177660 32025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727601e60f4d4738882153eac52205fd" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35721 } }
I20260806 03:42:11.177762 32025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:11.177809 32025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:11.177857 32025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 0 FOLLOWER]: Advancing to term 1
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:11.178447 32025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727601e60f4d4738882153eac52205fd" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35721 } }
I20260806 03:42:11.178576 32025 leader_election.cc:302] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727601e60f4d4738882153eac52205fd; no voters:
I20260806 03:42:11.178805 32025 leader_election.cc:288] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:11.178836 32032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:11.178964 32032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [term 1 LEADER]: Becoming Leader. State: Replica: 727601e60f4d4738882153eac52205fd, State: Running, Role: LEADER
I20260806 03:42:11.179054 32032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727601e60f4d4738882153eac52205fd" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35721 } }
I20260806 03:42:11.179113 32025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:11.179394 32033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727601e60f4d4738882153eac52205fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727601e60f4d4738882153eac52205fd" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35721 } } }
I20260806 03:42:11.179492 32033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:11.179674 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727601e60f4d4738882153eac52205fd. Latest consensus state: current_term: 1 leader_uuid: "727601e60f4d4738882153eac52205fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727601e60f4d4738882153eac52205fd" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35721 } } }
I20260806 03:42:11.179764 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:11.179713 32036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:11.180444 32036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:11.183013 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.172323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.30.24.62:35721)
0806 03:42:11.172558 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.172567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.172592 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:42:11.172998 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.173004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.173048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.173229 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.173899 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.174948 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.175767 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.176199 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.176324 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.178998 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.179054 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.179055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.179067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.181323 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.181774 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.181777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.181778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.181886 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.182146 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.182147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.182147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.182407 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.182565 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.182682 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.182836 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260806 03:42:11.184937 32036 catalog_manager.cc:1347] Generated new cluster ID: 231e8358e840491f91522900fa55509c
I20260806 03:42:11.185022 32036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:11.197773 32036 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:11.198514 32036 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:11.207239 32036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: Generated new TSK 0
I20260806 03:42:11.207473 32036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:11.243197 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35721
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:11.349843 32053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:11.350054 32053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:11.350088 32053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:11.351347 32053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:11.351397 32053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:11.351418 32053 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:11.351436 32053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:11.353056 32053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:11.353288 32053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:11.353495 32053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:11.358626 32059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.358560 32058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.358568 32061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:11.358777 32053 server_base.cc:1034] running on GCE node
I20260806 03:42:11.359066 32053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:11.359328 32053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:11.360522 32053 hybrid_clock.cc:648] HybridClock initialized: now 1785987731360507 us; error 30 us; skew 500 ppm
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:11.363118 32053 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:11.363572 32053 webserver.cc:460] Webserver started at http://127.30.24.1:46793/ using document root <none> and password file <none>
I20260806 03:42:11.363842 32053 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:11.363916 32053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:11.364065 32053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:11.365247 32053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "564886596e5443ecaac890daa51a947d"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "a29db553120f916fe7371e9724aa32b8"
server_key_iv: "21573b6da3c808aaa97f805d3649cc17"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.365620 32053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "564886596e5443ecaac890daa51a947d"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "a29db553120f916fe7371e9724aa32b8"
server_key_iv: "21573b6da3c808aaa97f805d3649cc17"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.367007 32053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:11.367767 32068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.367969 32053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.368083 32053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "564886596e5443ecaac890daa51a947d"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "a29db553120f916fe7371e9724aa32b8"
server_key_iv: "21573b6da3c808aaa97f805d3649cc17"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.368161 32053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:11.375376 32053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:11.377816 32053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:11.377998 32053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:11.378688 32053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:11.378744 32053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.378784 32053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:11.378804 32053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.386188 32053 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:44377
I20260806 03:42:11.386296 32187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:44377 every 8 connection(s)
I20260806 03:42:11.386631 32053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:11.389808 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 32053
I20260806 03:42:11.389927 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:11.390102 30816 external_mini_cluster.cc:1428] Setting key 88b79f793825bb45cd1d34bd0e801892
I20260806 03:42:11.390918 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.379572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:38259 (local address 127.30.24.62:35721)
0806 03:42:11.379645 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.379647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.380761 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 03:42:11.381318 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.381319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.381331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.381391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.382329 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.383167 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.384746 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.385096 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.385172 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.387788 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.387790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.387790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.387797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.389376 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.389798 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.389800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.389800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.389868 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.390160 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.390161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.390162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.390293 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.390373 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.390721 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.390819 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 03:42:11.391022 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.379235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.30.24.1:38259)
0806 03:42:11.380603 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.380660 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.381208 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.381468 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.381474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.381882 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.382235 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.382243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.383297 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.383298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.384657 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.384660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.385202 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.385204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.385391 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.385919 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.385932 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.387690 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.389449 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.389456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.389458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.389710 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.389941 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.389942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.389942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.390089 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.390365 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.390372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.390556 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.390672 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.390833 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":88,"threads_started":1}
I20260806 03:42:11.391548 32188 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35721
I20260806 03:42:11.391646 32188 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:11.391832 32188 heartbeater.cc:511] Master 127.30.24.62:35721 requested a full tablet report, sending...
I20260806 03:42:11.392361 31977 ts_manager.cc:194] Registered new tserver with Master: 564886596e5443ecaac890daa51a947d (127.30.24.1:44377)
I20260806 03:42:11.393150 31977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:38259
I20260806 03:42:11.398985 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.392807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:34241 (local address 127.30.24.62:35721)
0806 03:42:11.392841 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.392843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.393329 (+ 486us) server_negotiation.cc:407] Connection header received
0806 03:42:11.393413 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.393414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.393425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.393481 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.393903 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.394406 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.395083 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.395270 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.395330 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.396328 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.396329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.396329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.396335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.397752 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.398145 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.398146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.398147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.398207 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.398441 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.398441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.398442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.398551 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.398623 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.398811 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.398888 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:42:11.399000 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.392639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.30.24.1:34241)
0806 03:42:11.393257 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.393271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.393358 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.393486 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.393487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.393714 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.393800 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.393803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.394499 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.394500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.395007 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.395009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.395347 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.395348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.395436 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.395839 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.395844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.396243 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.397863 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.397865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.397866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.398052 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.398295 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.398296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.398297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.398365 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.398629 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.398630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.398678 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.398778 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.398890 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:11.421240 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35721
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:11.508711 32197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:11.508937 32197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:11.508965 32197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:11.510221 32197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:11.510277 32197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:11.510298 32197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:11.510358 32197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:11.511871 32197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:11.512156 32197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:11.512420 32197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:11.517963 32202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.517972 32204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:11.518095 32197 server_base.cc:1034] running on GCE node
W20260806 03:42:11.517959 32206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:11.518512 32197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:11.518790 32197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:11.519964 32197 hybrid_clock.cc:648] HybridClock initialized: now 1785987731519961 us; error 41 us; skew 500 ppm
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:11.522709 32197 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:11.523180 32197 webserver.cc:460] Webserver started at http://127.30.24.2:42469/ using document root <none> and password file <none>
I20260806 03:42:11.523423 32197 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:11.523495 32197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:11.523656 32197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:11.524845 32197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "5c44c98f73044e9588700a569e950761"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "1cb37fee1cc1d9de1863a70112c65544"
server_key_iv: "98954a7284682257d899db98edecc5fb"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.525267 32197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "5c44c98f73044e9588700a569e950761"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "1cb37fee1cc1d9de1863a70112c65544"
server_key_iv: "98954a7284682257d899db98edecc5fb"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.526651 32197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:11.527426 32214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.527602 32197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.527714 32197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "5c44c98f73044e9588700a569e950761"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "1cb37fee1cc1d9de1863a70112c65544"
server_key_iv: "98954a7284682257d899db98edecc5fb"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.527802 32197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:11.547631 32197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:11.550212 32197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:11.550398 32197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:11.550971 32197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:11.551040 32197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.551074 32197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:11.551185 32197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.558525 32197 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:35785
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:11.558627 32338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:35785 every 8 connection(s)
I20260806 03:42:11.558969 32197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:11.561990 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.551555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56949 (local address 127.30.24.62:35721)
0806 03:42:11.551605 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.551608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.551750 (+ 142us) server_negotiation.cc:407] Connection header received
0806 03:42:11.552327 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.552329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.552341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.552435 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.553421 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.554280 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.555902 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.556275 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.556363 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.559179 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.559181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.559181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.559188 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.560787 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.561166 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.561166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.561167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.561220 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.561444 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.561445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.561445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.561554 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.561606 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.561857 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.561924 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:42:11.562171 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.551446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.30.24.2:56949)
0806 03:42:11.551596 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.551639 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.552216 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.552567 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.552572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.552964 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.553303 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.553312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.554419 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.554420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.555795 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.555799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.556379 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.556379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.556559 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.557118 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.557132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.559023 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.560861 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.560867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.560869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.561095 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.561281 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.561282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.561282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.561387 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.561613 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.561618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.561743 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.561830 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.561959 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260806 03:42:11.562683 32339 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35721
I20260806 03:42:11.562774 32339 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:11.562976 32339 heartbeater.cc:511] Master 127.30.24.62:35721 requested a full tablet report, sending...
I20260806 03:42:11.563450 31976 ts_manager.cc:194] Registered new tserver with Master: 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785)
I20260806 03:42:11.564007 31976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:56949
I20260806 03:42:11.567354 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 32197
I20260806 03:42:11.567462 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:11.567700 30816 external_mini_cluster.cc:1428] Setting key 369955c436ebf3f432498d2b38ec7f6e
I20260806 03:42:11.569622 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.563881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:39267 (local address 127.30.24.62:35721)
0806 03:42:11.563922 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.563924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.563974 (+ 50us) server_negotiation.cc:407] Connection header received
0806 03:42:11.564049 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.564050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.564060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.564103 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.564543 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.565044 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.565756 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.565955 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.566030 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.566906 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.566907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.566908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.566915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.568431 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.568860 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.568861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.568861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.568914 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.569120 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.569120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.569121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.569266 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.569318 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.569492 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.569561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:11.569734 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.563751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.30.24.2:39267)
0806 03:42:11.563916 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.563929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.563992 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.564119 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.564120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.564336 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.564413 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.564415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.565148 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.565149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.565663 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.565664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.566029 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.566030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.566106 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.566522 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.566526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.566813 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.568538 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.568539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.568540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.568783 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.568981 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.568982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.568982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.569058 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.569331 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.569332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.569388 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.569461 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.569581 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:11.595623 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35721
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:11.680131 32349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:11.680415 32349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:11.680469 32349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:11.681725 32349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:11.681785 32349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:11.681806 32349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:11.681823 32349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:11.683208 32349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:11.683398 32349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:11.683566 32349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:11.688606 32354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.688617 32357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:11.688617 32355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:11.688823 32349 server_base.cc:1034] running on GCE node
I20260806 03:42:11.689146 32349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:11.689404 32349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:11.690569 32349 hybrid_clock.cc:648] HybridClock initialized: now 1785987731690556 us; error 30 us; skew 500 ppm
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:11.693202 32349 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:11.693631 32349 webserver.cc:460] Webserver started at http://127.30.24.3:43877/ using document root <none> and password file <none>
I20260806 03:42:11.693871 32349 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:11.693941 32349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:11.694098 32349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:11.695181 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "7a55c62b61904475b3d7ac2dfb69afe2"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "c4159f3dc6538f9802f76832163d1fea"
server_key_iv: "1d4770ab261a04c1e866c21dccef8b98"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.695573 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "7a55c62b61904475b3d7ac2dfb69afe2"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "c4159f3dc6538f9802f76832163d1fea"
server_key_iv: "1d4770ab261a04c1e866c21dccef8b98"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.696992 32349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:11.697729 32365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.697898 32349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:11.698005 32349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "7a55c62b61904475b3d7ac2dfb69afe2"
format_stamp: "Formatted at 2026-08-06 03:42:11 on dist-test-slave-10pc"
server_key: "c4159f3dc6538f9802f76832163d1fea"
server_key_iv: "1d4770ab261a04c1e866c21dccef8b98"
server_key_version: "encryptionkey@0"
I20260806 03:42:11.698081 32349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:11.705986 32349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:11.708106 32349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:11.708274 32349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:11.708830 32349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:11.708894 32349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.708936 32349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:11.708962 32349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:11.716267 32349 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:45941
I20260806 03:42:11.716327 32483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:45941 every 8 connection(s)
I20260806 03:42:11.716675 32349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:11.719684 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.709317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:46315 (local address 127.30.24.62:35721)
0806 03:42:11.709390 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.709392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.709571 (+ 179us) server_negotiation.cc:407] Connection header received
0806 03:42:11.710060 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.710062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.710074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.710170 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.710963 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.711776 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.713514 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.713728 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.713820 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.716707 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.716709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.716709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.716718 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.718242 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.718677 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.718679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.718679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.718744 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.719012 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.719013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.719013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.719141 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.719215 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.719502 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.719589 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 03:42:11.719879 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.709291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.30.24.3:46315)
0806 03:42:11.709440 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.709479 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.709965 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.710219 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.710223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.710569 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.710866 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.710873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.711898 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.711899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.713392 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.713396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.713844 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.713846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.713993 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.714658 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.714672 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.716597 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.718351 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.718359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.718361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.718588 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.718815 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.718816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.718816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.718935 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.719219 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.719227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.719369 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.719478 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.719650 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260806 03:42:11.720405 32484 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35721
I20260806 03:42:11.720508 32484 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:11.720708 32484 heartbeater.cc:511] Master 127.30.24.62:35721 requested a full tablet report, sending...
I20260806 03:42:11.721091 31977 ts_manager.cc:194] Registered new tserver with Master: 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941)
I20260806 03:42:11.721186 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 32349
I20260806 03:42:11.721273 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:11.721443 30816 external_mini_cluster.cc:1428] Setting key ee3fb517ec79a5b228dd42183c1735c0
I20260806 03:42:11.721597 31977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:46315
I20260806 03:42:11.722422 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:42:11.728405 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.721553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.30.24.3:40245)
0806 03:42:11.721791 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.721803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.721884 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.722596 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.722597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.722773 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.722835 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.722837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.723728 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.723728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.724223 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.724224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.724712 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.724713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.724802 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.725220 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.725226 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.725665 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.727319 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.727320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.727320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.727553 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.727749 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.727750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.727751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.727827 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.728058 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.728059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.728128 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.728214 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.728333 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":68,"threads_started":1}
I20260806 03:42:11.728492 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.721597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:40245 (local address 127.30.24.62:35721)
0806 03:42:11.721643 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.721646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.721855 (+ 209us) server_negotiation.cc:407] Connection header received
0806 03:42:11.722507 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.722508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.722521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.722579 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.722908 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.723637 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.724306 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.724627 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.724696 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.725758 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.725760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.725760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.725766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.727228 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.727631 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.727631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.727632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.727682 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.727898 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.727899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.727899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.727993 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.728040 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.728263 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.728348 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:11.747048 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.740128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.30.24.62:35721)
0806 03:42:11.740196 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.740199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.740211 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:11.740240 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.740241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.740258 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.740323 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.740970 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.741465 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.742199 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.742391 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.742468 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.744710 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.744712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.744712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.744720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.745913 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.746269 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.746271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.746271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.746330 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.746537 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.746538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.746539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.746652 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.746756 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.746862 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.746943 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:42:11.748631 31977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:11.750172 31977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:11.757063 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.752618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:45941 (local address 127.0.0.1:59068)
0806 03:42:11.753045 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.753056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.753137 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.753362 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.753366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.753368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.753443 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.753445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.754774 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.754776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.755830 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.755832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.756759 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.756760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.756842 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.756858 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.756934 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.756986 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260806 03:42:11.757071 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.752813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.30.24.3:45941)
0806 03:42:11.753011 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.753016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.753135 (+ 119us) server_negotiation.cc:407] Connection header received
0806 03:42:11.753212 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.753215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.753232 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.753526 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.753589 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.754692 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.755910 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.756666 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.756756 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.756812 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.756954 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.756988 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260806 03:42:11.757510 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.752644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44377 (local address 127.0.0.1:59548)
0806 03:42:11.753229 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.753239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.753312 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.753563 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.753564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.753566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.753628 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.753629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.754486 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.754487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.755783 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.755786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.757271 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.757272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.757347 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.757351 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.757398 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.757444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260806 03:42:11.757820 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.752880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.30.24.1:44377)
0806 03:42:11.753061 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.753066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.753286 (+ 220us) server_negotiation.cc:407] Connection header received
0806 03:42:11.753405 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.753407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.753424 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.753477 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.753700 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.754396 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.755884 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.756674 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.756749 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.756812 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.757677 (+ 865us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.757720 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260806 03:42:11.757838 32416 tablet_service.cc:1467] Processing CreateTablet for tablet f916d2dbb9674f469427e4ee7f4163dd (DEFAULT_TABLE table=test-table [id=513208c7107f4100b3b6843aa05ae71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:11.758011 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.752618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:35785 (local address 127.0.0.1:43794)
0806 03:42:11.752784 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.752826 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.752894 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.753371 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.753372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.753374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.753452 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.753454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.755918 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.755918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.756733 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.756736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.757771 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.757772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.757847 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.757851 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.757908 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.757952 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":45,"threads_started":1}
I20260806 03:42:11.758256 32416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f916d2dbb9674f469427e4ee7f4163dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.758313 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.752846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.30.24.2:35785)
0806 03:42:11.753138 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.753142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.753152 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:11.753195 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.753197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.753215 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.753298 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.753541 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.754680 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.756831 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.757601 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.757675 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.757736 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.758172 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.758216 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":93,"threads_started":1}
I20260806 03:42:11.758481 32111 tablet_service.cc:1467] Processing CreateTablet for tablet f916d2dbb9674f469427e4ee7f4163dd (DEFAULT_TABLE table=test-table [id=513208c7107f4100b3b6843aa05ae71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:11.758842 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f916d2dbb9674f469427e4ee7f4163dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.758886 32267 tablet_service.cc:1467] Processing CreateTablet for tablet f916d2dbb9674f469427e4ee7f4163dd (DEFAULT_TABLE table=test-table [id=513208c7107f4100b3b6843aa05ae71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:11.759223 32267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f916d2dbb9674f469427e4ee7f4163dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.761159 32508 tablet_bootstrap.cc:492] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Bootstrap starting.
I20260806 03:42:11.762063 32508 tablet_bootstrap.cc:654] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.762440 32508 log.cc:824] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:11.762888 32509 tablet_bootstrap.cc:492] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Bootstrap starting.
I20260806 03:42:11.763254 32511 tablet_bootstrap.cc:492] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Bootstrap starting.
I20260806 03:42:11.763392 32508 tablet_bootstrap.cc:492] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: No bootstrap required, opened a new log
I20260806 03:42:11.763463 32508 ts_tablet_manager.cc:1397] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:11.763615 32509 tablet_bootstrap.cc:654] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.763976 32511 tablet_bootstrap.cc:654] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.763990 32509 log.cc:824] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:11.764300 32511 log.cc:824] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:11.764801 32509 tablet_bootstrap.cc:492] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: No bootstrap required, opened a new log
I20260806 03:42:11.764895 32509 ts_tablet_manager.cc:1397] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:11.765116 32511 tablet_bootstrap.cc:492] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: No bootstrap required, opened a new log
I20260806 03:42:11.765192 32511 ts_tablet_manager.cc:1397] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 03:42:11.765714 32508 raft_consensus.cc:348] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.765895 32508 raft_consensus.cc:374] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.765929 32508 raft_consensus.cc:729] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a55c62b61904475b3d7ac2dfb69afe2, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.766073 32508 consensus_queue.cc:260] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.766358 32508 ts_tablet_manager.cc:1428] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:11.766496 32484 heartbeater.cc:503] Master 127.30.24.62:35721 was elected leader, sending a full tablet report...
I20260806 03:42:11.766983 32509 raft_consensus.cc:348] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.767127 32509 raft_consensus.cc:374] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.767171 32509 raft_consensus.cc:729] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564886596e5443ecaac890daa51a947d, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.767067 32511 raft_consensus.cc:348] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.767220 32511 raft_consensus.cc:374] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.767271 32511 raft_consensus.cc:729] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c44c98f73044e9588700a569e950761, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.767314 32509 consensus_queue.cc:260] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.767383 32511 consensus_queue.cc:260] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.767668 32339 heartbeater.cc:503] Master 127.30.24.62:35721 was elected leader, sending a full tablet report...
I20260806 03:42:11.767664 32509 ts_tablet_manager.cc:1428] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:11.767845 32188 heartbeater.cc:503] Master 127.30.24.62:35721 was elected leader, sending a full tablet report...
I20260806 03:42:11.767935 32511 ts_tablet_manager.cc:1428] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 03:42:11.793846 32516 raft_consensus.cc:482] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:11.794018 32516 raft_consensus.cc:504] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.794430 32516 leader_election.cc:288] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785), 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941)
I20260806 03:42:11.798373 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.794660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:32773 (local address 127.30.24.2:35785)
0806 03:42:11.794709 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.794712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.794724 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:11.794755 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.794756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.794767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.794835 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.795013 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.795802 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.797409 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.797975 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.798038 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.798049 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.798255 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.798298 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:11.798373 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.794698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:32893 (local address 127.30.24.3:45941)
0806 03:42:11.794750 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.794751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.794901 (+ 150us) server_negotiation.cc:407] Connection header received
0806 03:42:11.794931 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.794932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.794949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.795015 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.795168 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.795867 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.797411 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.797955 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.798038 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.798049 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.798254 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.798299 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:42:11.798400 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.794585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:35785 (local address 127.30.24.1:32773)
0806 03:42:11.794642 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.794655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.794702 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.794850 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.794851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.794853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.794934 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.794937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.795920 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.795921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.797305 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.797309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.798060 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.798061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.798164 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.798171 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.798234 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.798277 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":46}
I20260806 03:42:11.798411 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.794593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:45941 (local address 127.30.24.1:32893)
0806 03:42:11.794815 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.794824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.794878 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.795032 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.795033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.795034 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.795100 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.795100 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.795937 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.795938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.797305 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.797309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.798060 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.798061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.798164 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.798171 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.798234 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.798277 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":123,"thread_start_us":122,"threads_started":1}
I20260806 03:42:11.798799 32289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f916d2dbb9674f469427e4ee7f4163dd" candidate_uuid: "564886596e5443ecaac890daa51a947d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c44c98f73044e9588700a569e950761" is_pre_election: true
I20260806 03:42:11.798825 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f916d2dbb9674f469427e4ee7f4163dd" candidate_uuid: "564886596e5443ecaac890daa51a947d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" is_pre_election: true
I20260806 03:42:11.798988 32436 raft_consensus.cc:2454] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564886596e5443ecaac890daa51a947d in term 0.
I20260806 03:42:11.798980 32289 raft_consensus.cc:2454] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564886596e5443ecaac890daa51a947d in term 0.
I20260806 03:42:11.799237 32070 leader_election.cc:302] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 564886596e5443ecaac890daa51a947d, 7a55c62b61904475b3d7ac2dfb69afe2; no voters:
I20260806 03:42:11.799427 32516 raft_consensus.cc:2781] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:11.799487 32516 raft_consensus.cc:482] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:11.799515 32516 raft_consensus.cc:3037] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:11.800513 32516 raft_consensus.cc:504] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.800683 32516 leader_election.cc:288] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [CANDIDATE]: Term 1 election: Requested vote from peers 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785), 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941)
I20260806 03:42:11.800838 32289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f916d2dbb9674f469427e4ee7f4163dd" candidate_uuid: "564886596e5443ecaac890daa51a947d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c44c98f73044e9588700a569e950761"
I20260806 03:42:11.800860 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f916d2dbb9674f469427e4ee7f4163dd" candidate_uuid: "564886596e5443ecaac890daa51a947d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a55c62b61904475b3d7ac2dfb69afe2"
I20260806 03:42:11.800920 32289 raft_consensus.cc:3037] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:11.800977 32436 raft_consensus.cc:3037] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:11.801712 32289 raft_consensus.cc:2454] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564886596e5443ecaac890daa51a947d in term 1.
I20260806 03:42:11.801898 32072 leader_election.cc:302] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 564886596e5443ecaac890daa51a947d, 5c44c98f73044e9588700a569e950761; no voters:
I20260806 03:42:11.801906 32436 raft_consensus.cc:2454] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564886596e5443ecaac890daa51a947d in term 1.
I20260806 03:42:11.801985 32516 raft_consensus.cc:2781] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:11.802175 32516 raft_consensus.cc:686] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 1 LEADER]: Becoming Leader. State: Replica: 564886596e5443ecaac890daa51a947d, State: Running, Role: LEADER
I20260806 03:42:11.802289 32516 consensus_queue.cc:237] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } }
I20260806 03:42:11.803243 31975 catalog_manager.cc:5495] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d reported cstate change: term changed from 0 to 1, leader changed from <none> to 564886596e5443ecaac890daa51a947d (127.30.24.1). New cstate: current_term: 1 leader_uuid: "564886596e5443ecaac890daa51a947d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } health_report { overall_health: HEALTHY } } }
W20260806 03:42:11.809562 32340 tablet.cc:2369] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:11.821164 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.818468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.30.24.1:44377)
0806 03:42:11.818531 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.818533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.818543 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:11.818635 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.818638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.818674 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.818746 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:11.818902 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.819400 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.820230 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.820425 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.820549 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.820793 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:11.820893 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:11.820980 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.821026 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.821048 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 03:42:11.823136 32289 raft_consensus.cc:1261] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Refusing update from remote peer 564886596e5443ecaac890daa51a947d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:11.823136 32436 raft_consensus.cc:1261] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Refusing update from remote peer 564886596e5443ecaac890daa51a947d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:11.823575 32518 consensus_queue.cc:1035] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:11.823711 32516 consensus_queue.cc:1035] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:11.826867 32521 mvcc.cc:205] Tried to move back new op lower bound from 7315405749544136704 to 7315405749462974464. Current Snapshot: MvccSnapshot[applied={T|T < 7315405749544136704 or (T in {7315405749544136704})}]
I20260806 03:42:11.826993 32522 mvcc.cc:205] Tried to move back new op lower bound from 7315405749544136704 to 7315405749462974464. Current Snapshot: MvccSnapshot[applied={T|T < 7315405749544136704}]
I20260806 03:42:11.830255 31975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:11.830367 31975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:11.831350 31975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: Sending DeleteTablet for 3 replicas of tablet f916d2dbb9674f469427e4ee7f4163dd
I20260806 03:42:11.831869 32416 tablet_service.cc:1514] Processing DeleteTablet for tablet f916d2dbb9674f469427e4ee7f4163dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:11 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59068
I20260806 03:42:11.831877 32267 tablet_service.cc:1514] Processing DeleteTablet for tablet f916d2dbb9674f469427e4ee7f4163dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:11 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:43794
I20260806 03:42:11.832144 32531 tablet_replica.cc:330] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: stopping tablet replica
I20260806 03:42:11.832152 32111 tablet_service.cc:1514] Processing DeleteTablet for tablet f916d2dbb9674f469427e4ee7f4163dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:11 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59548
I20260806 03:42:11.832257 32531 raft_consensus.cc:2229] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:11.832405 32532 tablet_replica.cc:330] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: stopping tablet replica
I20260806 03:42:11.832572 32532 raft_consensus.cc:2229] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:11.832659 32531 raft_consensus.cc:2258] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:11.832634 31975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:11.832770 31975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:11.832935 32532 raft_consensus.cc:2258] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:11.832981 32530 tablet_replica.cc:330] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: stopping tablet replica
I20260806 03:42:11.833076 32530 raft_consensus.cc:2229] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:11.833524 32531 ts_tablet_manager.cc:1905] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:11.833537 32532 ts_tablet_manager.cc:1905] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:11.833192 32530 raft_consensus.cc:2258] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:11.834370 32111 tablet_service.cc:1467] Processing CreateTablet for tablet 41f9edd40ab745aba5d1e8dfc7585fa3 (DEFAULT_TABLE table=test-table [id=fd928cc869764fb8a0430980596cfc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:11.834494 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f9edd40ab745aba5d1e8dfc7585fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.834899 32416 tablet_service.cc:1467] Processing CreateTablet for tablet 41f9edd40ab745aba5d1e8dfc7585fa3 (DEFAULT_TABLE table=test-table [id=fd928cc869764fb8a0430980596cfc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:11.835033 32416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f9edd40ab745aba5d1e8dfc7585fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.835834 32509 tablet_bootstrap.cc:492] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Bootstrap starting.
I20260806 03:42:11.836270 32509 tablet_bootstrap.cc:654] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.836505 32508 tablet_bootstrap.cc:492] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Bootstrap starting.
I20260806 03:42:11.836791 32267 tablet_service.cc:1467] Processing CreateTablet for tablet 41f9edd40ab745aba5d1e8dfc7585fa3 (DEFAULT_TABLE table=test-table [id=fd928cc869764fb8a0430980596cfc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:11.836918 32267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f9edd40ab745aba5d1e8dfc7585fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.836925 32508 tablet_bootstrap.cc:654] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.837160 32530 ts_tablet_manager.cc:1905] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:11.837756 32509 tablet_bootstrap.cc:492] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: No bootstrap required, opened a new log
I20260806 03:42:11.837801 32509 ts_tablet_manager.cc:1397] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:11.837952 32509 raft_consensus.cc:348] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.838018 32509 raft_consensus.cc:374] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.838042 32509 raft_consensus.cc:729] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564886596e5443ecaac890daa51a947d, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.838088 32509 consensus_queue.cc:260] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.838169 32509 ts_tablet_manager.cc:1428] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.838377 32511 tablet_bootstrap.cc:492] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Bootstrap starting.
I20260806 03:42:11.838845 32508 tablet_bootstrap.cc:492] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: No bootstrap required, opened a new log
I20260806 03:42:11.838892 32508 ts_tablet_manager.cc:1397] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:11.838888 32511 tablet_bootstrap.cc:654] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.839041 32508 raft_consensus.cc:348] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.839107 32508 raft_consensus.cc:374] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.839129 32508 raft_consensus.cc:729] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a55c62b61904475b3d7ac2dfb69afe2, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.839171 32508 consensus_queue.cc:260] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.839246 32508 ts_tablet_manager.cc:1428] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.839715 32532 ts_tablet_manager.cc:1918] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:11.839838 32532 log.cc:1196] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/f916d2dbb9674f469427e4ee7f4163dd
I20260806 03:42:11.839951 32531 ts_tablet_manager.cc:1918] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:11.840027 32531 log.cc:1196] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/f916d2dbb9674f469427e4ee7f4163dd
I20260806 03:42:11.840268 32532 ts_tablet_manager.cc:1939] T f916d2dbb9674f469427e4ee7f4163dd P 564886596e5443ecaac890daa51a947d: Deleting consensus metadata
I20260806 03:42:11.840279 32531 ts_tablet_manager.cc:1939] T f916d2dbb9674f469427e4ee7f4163dd P 5c44c98f73044e9588700a569e950761: Deleting consensus metadata
I20260806 03:42:11.840715 31964 catalog_manager.cc:4841] TS 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785): tablet f916d2dbb9674f469427e4ee7f4163dd (table test-table [id=513208c7107f4100b3b6843aa05ae71f]) successfully deleted
I20260806 03:42:11.840797 31961 catalog_manager.cc:4841] TS 564886596e5443ecaac890daa51a947d (127.30.24.1:44377): tablet f916d2dbb9674f469427e4ee7f4163dd (table test-table [id=513208c7107f4100b3b6843aa05ae71f]) successfully deleted
I20260806 03:42:11.840891 32530 ts_tablet_manager.cc:1918] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:11.840969 32530 log.cc:1196] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/f916d2dbb9674f469427e4ee7f4163dd
I20260806 03:42:11.841246 32530 ts_tablet_manager.cc:1939] T f916d2dbb9674f469427e4ee7f4163dd P 7a55c62b61904475b3d7ac2dfb69afe2: Deleting consensus metadata
I20260806 03:42:11.841316 32511 tablet_bootstrap.cc:492] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: No bootstrap required, opened a new log
I20260806 03:42:11.841384 32511 ts_tablet_manager.cc:1397] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 03:42:11.841538 32511 raft_consensus.cc:348] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.841607 32511 raft_consensus.cc:374] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.841629 32511 raft_consensus.cc:729] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c44c98f73044e9588700a569e950761, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.841670 32511 consensus_queue.cc:260] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.841717 31962 catalog_manager.cc:4841] TS 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941): tablet f916d2dbb9674f469427e4ee7f4163dd (table test-table [id=513208c7107f4100b3b6843aa05ae71f]) successfully deleted
I20260806 03:42:11.841753 32511 ts_tablet_manager.cc:1428] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260806 03:42:11.887437 32189 tablet.cc:2369] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:11.903015 32515 raft_consensus.cc:482] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:11.903141 32515 raft_consensus.cc:504] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.903527 32515 leader_election.cc:288] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 564886596e5443ecaac890daa51a947d (127.30.24.1:44377), 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941)
I20260806 03:42:11.907519 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.903939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48297 (local address 127.30.24.3:45941)
0806 03:42:11.904023 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.904026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.904036 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:11.904071 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.904072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.904083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.904159 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.904388 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.905013 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.906463 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.907116 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.907216 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.907229 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.907398 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.907445 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 03:42:11.907554 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.903747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:45941 (local address 127.30.24.2:48297)
0806 03:42:11.903792 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.903805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.903856 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.904193 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.904195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.904198 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.904280 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.904282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.905118 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.905119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.906353 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.906356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.907213 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.907214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.907300 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.907303 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.907387 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.907435 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 03:42:11.907855 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f9edd40ab745aba5d1e8dfc7585fa3" candidate_uuid: "5c44c98f73044e9588700a569e950761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" is_pre_election: true
I20260806 03:42:11.907953 32436 raft_consensus.cc:2454] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c44c98f73044e9588700a569e950761 in term 0.
I20260806 03:42:11.908095 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.903842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:49405 (local address 127.30.24.1:44377)
0806 03:42:11.903895 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.903897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.904164 (+ 267us) server_negotiation.cc:407] Connection header received
0806 03:42:11.904285 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.904287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.904303 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.904373 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:11.905295 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.905791 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.907113 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.907671 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.907761 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.907772 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.907983 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.908025 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 03:42:11.908157 32216 leader_election.cc:302] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c44c98f73044e9588700a569e950761, 7a55c62b61904475b3d7ac2dfb69afe2; no voters:
I20260806 03:42:11.908340 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.903747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44377 (local address 127.30.24.2:49405)
0806 03:42:11.904097 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.904108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.904197 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.905120 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.905121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.905123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:11.905204 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.905205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.905889 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.905890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.906990 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.906992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.907812 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.907813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.907896 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.907900 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.908192 (+ 292us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.908242 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":202,"thread_start_us":67,"threads_started":1}
I20260806 03:42:11.908342 32515 raft_consensus.cc:2781] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:11.908459 32515 raft_consensus.cc:482] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:11.908521 32515 raft_consensus.cc:3037] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:11.908695 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f9edd40ab745aba5d1e8dfc7585fa3" candidate_uuid: "5c44c98f73044e9588700a569e950761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564886596e5443ecaac890daa51a947d" is_pre_election: true
I20260806 03:42:11.908850 32140 raft_consensus.cc:2454] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c44c98f73044e9588700a569e950761 in term 0.
I20260806 03:42:11.909857 32515 raft_consensus.cc:504] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.910009 32515 leader_election.cc:288] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [CANDIDATE]: Term 1 election: Requested vote from peers 564886596e5443ecaac890daa51a947d (127.30.24.1:44377), 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941)
I20260806 03:42:11.910182 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f9edd40ab745aba5d1e8dfc7585fa3" candidate_uuid: "5c44c98f73044e9588700a569e950761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564886596e5443ecaac890daa51a947d"
I20260806 03:42:11.910197 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f9edd40ab745aba5d1e8dfc7585fa3" candidate_uuid: "5c44c98f73044e9588700a569e950761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a55c62b61904475b3d7ac2dfb69afe2"
I20260806 03:42:11.910267 32436 raft_consensus.cc:3037] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:11.910267 32140 raft_consensus.cc:3037] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:11.911062 32140 raft_consensus.cc:2454] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c44c98f73044e9588700a569e950761 in term 1.
I20260806 03:42:11.911062 32436 raft_consensus.cc:2454] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c44c98f73044e9588700a569e950761 in term 1.
I20260806 03:42:11.911248 32216 leader_election.cc:302] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c44c98f73044e9588700a569e950761, 7a55c62b61904475b3d7ac2dfb69afe2; no voters:
I20260806 03:42:11.911346 32515 raft_consensus.cc:2781] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:11.911516 32515 raft_consensus.cc:686] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 1 LEADER]: Becoming Leader. State: Replica: 5c44c98f73044e9588700a569e950761, State: Running, Role: LEADER
I20260806 03:42:11.911597 32515 consensus_queue.cc:237] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.912384 31975 catalog_manager.cc:5495] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c44c98f73044e9588700a569e950761 (127.30.24.2). New cstate: current_term: 1 leader_uuid: "5c44c98f73044e9588700a569e950761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } health_report { overall_health: UNKNOWN } } }
Aug 06 03:42:11 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:11.960170 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.953328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.30.24.62:35721)
0806 03:42:11.953401 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:11.953403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:11.953573 (+ 170us) server_negotiation.cc:407] Connection header received
0806 03:42:11.953639 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:11.953640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:11.953649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:11.953702 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:11.954328 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.954830 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.955560 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.955741 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.955790 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.957946 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:11.957947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:11.957947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:11.957954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:11.959159 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.959548 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.959549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.959549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.959600 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:11.959775 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:11.959775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:11.959775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:11.959870 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:11.959914 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:11.960035 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 03:42:11.960100 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:42:11.960317 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:11.953275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35721 (local address 127.0.0.1:59862)
0806 03:42:11.953479 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:11.953493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:11.953582 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:11.953731 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:11.953732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:11.954190 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:11.954255 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.954257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.954902 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.954903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.955483 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:11.955486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:11.955806 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:11.955807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:11.955850 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:11.956441 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:11.956446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:11.957867 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:11.959253 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.959261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.959263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.959476 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.959669 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:11.959670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:11.959670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:11.959729 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:11.959919 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:11.959926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:11.959968 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:42:11.960020 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 03:42:11.960152 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":47,"thread_start_us":116,"threads_started":1}
I20260806 03:42:11.962524 31975 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59862:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 03:42:11.962813 31975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:11.964395 32416 tablet_service.cc:1467] Processing CreateTablet for tablet 8be5a688337b496dae53b8f20b938439 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ab36fd4db1c413b8b4e3d87631ba990]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:11.964395 32267 tablet_service.cc:1467] Processing CreateTablet for tablet 8be5a688337b496dae53b8f20b938439 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ab36fd4db1c413b8b4e3d87631ba990]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:11.964346 32111 tablet_service.cc:1467] Processing CreateTablet for tablet 8be5a688337b496dae53b8f20b938439 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ab36fd4db1c413b8b4e3d87631ba990]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:11.964572 32416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8be5a688337b496dae53b8f20b938439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.964572 32267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8be5a688337b496dae53b8f20b938439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.964591 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8be5a688337b496dae53b8f20b938439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:11.967327 32509 tablet_bootstrap.cc:492] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: Bootstrap starting.
I20260806 03:42:11.967368 32511 tablet_bootstrap.cc:492] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: Bootstrap starting.
I20260806 03:42:11.967357 32508 tablet_bootstrap.cc:492] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: Bootstrap starting.
I20260806 03:42:11.967748 32509 tablet_bootstrap.cc:654] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.967806 32511 tablet_bootstrap.cc:654] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:11.967849 32508 tablet_bootstrap.cc:654] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:11.968114 32485 tablet.cc:2369] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:11.968642 32511 tablet_bootstrap.cc:492] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: No bootstrap required, opened a new log
I20260806 03:42:11.968708 32511 ts_tablet_manager.cc:1397] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:11.968894 32508 tablet_bootstrap.cc:492] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: No bootstrap required, opened a new log
I20260806 03:42:11.968873 32511 raft_consensus.cc:348] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.968937 32508 ts_tablet_manager.cc:1397] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:11.968953 32511 raft_consensus.cc:374] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.968986 32511 raft_consensus.cc:729] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c44c98f73044e9588700a569e950761, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.969043 32511 consensus_queue.cc:260] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.969090 32508 raft_consensus.cc:348] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.969156 32511 ts_tablet_manager.cc:1428] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:11.969183 32508 raft_consensus.cc:374] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.969206 32508 raft_consensus.cc:729] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a55c62b61904475b3d7ac2dfb69afe2, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.969256 32508 consensus_queue.cc:260] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.969221 32515 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:11.969317 32515 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:11.969352 32508 ts_tablet_manager.cc:1428] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.969436 32509 tablet_bootstrap.cc:492] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: No bootstrap required, opened a new log
I20260806 03:42:11.969509 32509 ts_tablet_manager.cc:1397] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:11.969607 32514 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:11.969655 32509 raft_consensus.cc:348] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.969731 32509 raft_consensus.cc:374] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:11.969759 32509 raft_consensus.cc:729] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564886596e5443ecaac890daa51a947d, State: Initialized, Role: FOLLOWER
I20260806 03:42:11.969743 32514 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:11.969812 32509 consensus_queue.cc:260] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:11.969920 32509 ts_tablet_manager.cc:1428] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:11.969939 32518 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:11.970022 32518 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:12.122251 32514 raft_consensus.cc:482] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:12.122360 32514 raft_consensus.cc:504] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:12.122717 32514 leader_election.cc:288] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 564886596e5443ecaac890daa51a947d (127.30.24.1:44377), 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785)
I20260806 03:42:12.125581 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.122991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:50329 (local address 127.30.24.2:35785)
0806 03:42:12.123035 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.123037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.123047 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:12.123077 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.123078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.123095 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.123163 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:12.123337 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.123823 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.124839 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.125219 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.125303 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.125315 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.125467 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.125507 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:12.125581 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.122891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:35785 (local address 127.30.24.3:50329)
0806 03:42:12.122951 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:12.122963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:12.123010 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:12.123169 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:12.123171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:12.123172 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:12.123264 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.123265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.123914 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.123915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.124748 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.124751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.125315 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.125317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.125391 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.125395 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:12.125460 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 03:42:12.125506 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 03:42:12.125849 32289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a688337b496dae53b8f20b938439" candidate_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c44c98f73044e9588700a569e950761" is_pre_election: true
I20260806 03:42:12.125967 32289 raft_consensus.cc:2454] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a55c62b61904475b3d7ac2dfb69afe2 in term 0.
I20260806 03:42:12.126168 32369 leader_election.cc:302] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c44c98f73044e9588700a569e950761, 7a55c62b61904475b3d7ac2dfb69afe2; no voters:
I20260806 03:42:12.126330 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.122957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:45873 (local address 127.30.24.1:44377)
0806 03:42:12.123011 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.123014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.123194 (+ 180us) server_negotiation.cc:407] Connection header received
0806 03:42:12.123269 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.123270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.123283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.123358 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:12.123519 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.123979 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.125442 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.125942 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.126019 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.126031 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.126201 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.126238 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:42:12.126317 32514 raft_consensus.cc:2781] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:12.126379 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.122891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44377 (local address 127.30.24.3:45873)
0806 03:42:12.123100 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:12.123127 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:12.123210 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:12.123359 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:12.123360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:12.123362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:12.123448 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.123449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.124073 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.124074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.125342 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.125345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.126035 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.126037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.126116 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.126121 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:12.126183 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:42:12.126225 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":15,"thread_start_us":138,"threads_started":1}
I20260806 03:42:12.126381 32514 raft_consensus.cc:482] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:12.126442 32514 raft_consensus.cc:3037] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:12.126685 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a688337b496dae53b8f20b938439" candidate_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564886596e5443ecaac890daa51a947d" is_pre_election: true
I20260806 03:42:12.126793 32140 raft_consensus.cc:2454] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a55c62b61904475b3d7ac2dfb69afe2 in term 0.
I20260806 03:42:12.127278 32514 raft_consensus.cc:504] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:12.127432 32514 leader_election.cc:288] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [CANDIDATE]: Term 1 election: Requested vote from peers 564886596e5443ecaac890daa51a947d (127.30.24.1:44377), 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785)
I20260806 03:42:12.127581 32289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a688337b496dae53b8f20b938439" candidate_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c44c98f73044e9588700a569e950761"
I20260806 03:42:12.127597 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a688337b496dae53b8f20b938439" candidate_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564886596e5443ecaac890daa51a947d"
I20260806 03:42:12.127662 32140 raft_consensus.cc:3037] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:12.127662 32289 raft_consensus.cc:3037] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:12.128409 32140 raft_consensus.cc:2454] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a55c62b61904475b3d7ac2dfb69afe2 in term 1.
I20260806 03:42:12.128409 32289 raft_consensus.cc:2454] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a55c62b61904475b3d7ac2dfb69afe2 in term 1.
I20260806 03:42:12.128605 32366 leader_election.cc:302] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 564886596e5443ecaac890daa51a947d, 7a55c62b61904475b3d7ac2dfb69afe2; no voters:
I20260806 03:42:12.128705 32514 raft_consensus.cc:2781] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:12.128845 32514 raft_consensus.cc:686] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 LEADER]: Becoming Leader. State: Replica: 7a55c62b61904475b3d7ac2dfb69afe2, State: Running, Role: LEADER
I20260806 03:42:12.128901 32514 consensus_queue.cc:237] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } }
I20260806 03:42:12.129289 32550 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: TxnStatusTablet state changed. Reason: New leader 7a55c62b61904475b3d7ac2dfb69afe2. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.129410 32550 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:12.129662 32552 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 03:42:12.129786 32552 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 03:42:12.129711 31976 catalog_manager.cc:5495] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3). New cstate: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:12.151952 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.149428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.30.24.3:45941)
0806 03:42:12.149499 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.149501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.149512 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:12.149600 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.149601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.149637 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.149711 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:12.149858 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.150350 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.151104 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.151306 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.151387 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.151598 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.151707 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.151792 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.151850 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.151870 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:42:12.151964 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.149356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:45941 (local address 127.0.0.1:59078)
0806 03:42:12.149438 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:12.149453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:12.149508 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:12.149720 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:12.149721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:12.149722 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:12.149774 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.149781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.150439 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.150441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.151010 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.151011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.151388 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.151390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.151471 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.151487 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.151776 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.151788 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 03:42:12.151845 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:42:12.151898 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 03:42:12.153338 32289 raft_consensus.cc:1261] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Refusing update from remote peer 7a55c62b61904475b3d7ac2dfb69afe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:12.153643 32550 consensus_queue.cc:1035] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:12.153716 32140 raft_consensus.cc:1261] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Refusing update from remote peer 7a55c62b61904475b3d7ac2dfb69afe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:12.153945 32550 consensus_queue.cc:1035] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:12.154655 32518 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: TxnStatusTablet state changed. Reason: New leader 7a55c62b61904475b3d7ac2dfb69afe2. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.154754 32518 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:12.154837 32515 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: TxnStatusTablet state changed. Reason: New leader 7a55c62b61904475b3d7ac2dfb69afe2. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.154929 32515 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:12.155373 32514 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.155467 32514 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:12.155568 32514 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.155632 32514 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 7a55c62b61904475b3d7ac2dfb69afe2: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:12.155997 32518 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.156086 32518 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 564886596e5443ecaac890daa51a947d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:12.156306 32523 mvcc.cc:205] Tried to move back new op lower bound from 7315405750897860608 to 7315405750800576512. Current Snapshot: MvccSnapshot[applied={T|T < 7315405750897860608}]
I20260806 03:42:12.156390 32515 tablet_replica.cc:439] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 } } peers { permanent_uuid: "5c44c98f73044e9588700a569e950761" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35785 } } peers { permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 } } }
I20260806 03:42:12.156445 32515 tablet_replica.cc:442] T 8be5a688337b496dae53b8f20b938439 P 5c44c98f73044e9588700a569e950761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:12.160543 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.158032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.30.24.2:35785)
0806 03:42:12.158076 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.158079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.158088 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:12.158174 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.158174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.158193 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.158280 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:12.158385 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.158960 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.159700 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.159908 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.159981 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.160170 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.160281 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.160362 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.160407 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.160427 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:42:12.166033 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.163240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:57735 (local address 127.30.24.3:45941)
0806 03:42:12.163280 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.163282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.163425 (+ 143us) server_negotiation.cc:407] Connection header received
0806 03:42:12.163480 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.163480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.163489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.163537 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:12.163730 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.164235 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.165380 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.165561 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.165622 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.165820 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.165847 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.165881 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.165960 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.165976 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:42:12.166098 32560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.163171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:45941 (local address 127.30.24.2:57735)
0806 03:42:12.163354 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:12.163365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:12.163424 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:12.163580 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:12.163581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:12.163582 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:12.163654 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.163656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.164313 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.164313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.165294 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.165296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.165641 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.165642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.165714 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.165732 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.165902 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.165904 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:12.165959 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 03:42:12.166004 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260806 03:42:12.168529 32436 raft_consensus.cc:1261] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Refusing update from remote peer 5c44c98f73044e9588700a569e950761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:12.168533 32140 raft_consensus.cc:1261] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Refusing update from remote peer 5c44c98f73044e9588700a569e950761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:12.168795 32515 consensus_queue.cc:1035] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "564886596e5443ecaac890daa51a947d" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:12.169165 32537 consensus_queue.cc:1035] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a55c62b61904475b3d7ac2dfb69afe2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:12.178985 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.176174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:35785 (local address 127.30.24.3:43557)
0806 03:42:12.176244 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:12.176255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:12.176291 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:12.176416 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:12.176417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:12.176419 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:12.176500 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.176502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.177205 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.177206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.178142 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:12.178145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.178608 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.178609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:12.178671 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.178687 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.178862 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.178864 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:12.178897 (+ 33us) client_negotiation.cc:240] Negotiation successful
0806 03:42:12.178932 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 03:42:12.178996 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.176233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:43557 (local address 127.30.24.2:35785)
0806 03:42:12.176274 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.176276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.176285 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:12.176358 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.176359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.176368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.176413 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:12.176557 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.177130 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.178242 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.178536 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.178610 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.178768 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:12.178801 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:12.178855 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.178903 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.178923 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:42:12.186664 31975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:12.186791 31975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:12.187511 31975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 727601e60f4d4738882153eac52205fd: Sending DeleteTablet for 3 replicas of tablet 41f9edd40ab745aba5d1e8dfc7585fa3
I20260806 03:42:12.187819 32111 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f9edd40ab745aba5d1e8dfc7585fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:12 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59548
I20260806 03:42:12.187932 32532 tablet_replica.cc:330] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: stopping tablet replica
I20260806 03:42:12.187992 32532 raft_consensus.cc:2229] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:12.188037 32532 raft_consensus.cc:2258] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:12.188270 32267 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f9edd40ab745aba5d1e8dfc7585fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:12 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:43794
I20260806 03:42:12.188366 32531 tablet_replica.cc:330] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: stopping tablet replica
I20260806 03:42:12.188351 32532 ts_tablet_manager.cc:1905] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:12.188410 32531 raft_consensus.cc:2229] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:12.188633 32531 raft_consensus.cc:2258] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:12.188977 32531 ts_tablet_manager.cc:1905] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:12.190106 32532 ts_tablet_manager.cc:1918] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:12.190163 32532 log.cc:1196] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/41f9edd40ab745aba5d1e8dfc7585fa3
I20260806 03:42:12.190382 32532 ts_tablet_manager.cc:1939] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 564886596e5443ecaac890daa51a947d: Deleting consensus metadata
I20260806 03:42:12.190842 32531 ts_tablet_manager.cc:1918] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:12.190915 32531 log.cc:1196] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/41f9edd40ab745aba5d1e8dfc7585fa3
I20260806 03:42:12.191148 32531 ts_tablet_manager.cc:1939] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 5c44c98f73044e9588700a569e950761: Deleting consensus metadata
I20260806 03:42:12.191507 31964 catalog_manager.cc:4841] TS 5c44c98f73044e9588700a569e950761 (127.30.24.2:35785): tablet 41f9edd40ab745aba5d1e8dfc7585fa3 (table test-table [id=fd928cc869764fb8a0430980596cfc9f]) successfully deleted
I20260806 03:42:12.192324 32416 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f9edd40ab745aba5d1e8dfc7585fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:12 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59068
I20260806 03:42:12.192435 32530 tablet_replica.cc:330] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: stopping tablet replica
I20260806 03:42:12.192518 32530 raft_consensus.cc:2229] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:12.192559 32530 raft_consensus.cc:2258] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:12.192734 31961 catalog_manager.cc:4841] TS 564886596e5443ecaac890daa51a947d (127.30.24.1:44377): tablet 41f9edd40ab745aba5d1e8dfc7585fa3 (table test-table [id=fd928cc869764fb8a0430980596cfc9f]) successfully deleted
I20260806 03:42:12.192876 32530 ts_tablet_manager.cc:1905] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:12.194595 32530 ts_tablet_manager.cc:1918] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:12.194689 32530 log.cc:1196] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/41f9edd40ab745aba5d1e8dfc7585fa3
I20260806 03:42:12.194927 32530 ts_tablet_manager.cc:1939] T 41f9edd40ab745aba5d1e8dfc7585fa3 P 7a55c62b61904475b3d7ac2dfb69afe2: Deleting consensus metadata
I20260806 03:42:12.195288 31962 catalog_manager.cc:4841] TS 7a55c62b61904475b3d7ac2dfb69afe2 (127.30.24.3:45941): tablet 41f9edd40ab745aba5d1e8dfc7585fa3 (table test-table [id=fd928cc869764fb8a0430980596cfc9f]) successfully deleted
Aug 06 03:42:12 dist-test-slave-10pc krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:12.207827 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.192378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.30.24.1:44377)
0806 03:42:12.192480 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.192482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.196644 (+ 4162us) server_negotiation.cc:407] Connection header received
0806 03:42:12.196766 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.196767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.196781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.196863 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:12.197753 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.198598 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.199687 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.200030 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.200116 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.204199 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:12.204212 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:12.204213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:12.204222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:12.206374 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:12.206841 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:12.206845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:12.206845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:12.206917 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:12.207164 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:12.207165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:12.207166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:12.207358 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:12.207529 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.207634 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.207726 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 03:42:12.208444 32182 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59580
I20260806 03:42:12.218185 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:12.209642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.30.24.62:35721)
0806 03:42:12.209692 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:12.209694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:12.209975 (+ 281us) server_negotiation.cc:407] Connection header received
0806 03:42:12.210046 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:12.210047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:12.210060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:12.210135 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:12.210901 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.211682 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.212917 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:12.213235 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:12.213319 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:12.214882 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:12.214884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:12.214884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:12.214892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:12.216790 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:12.217238 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:12.217239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:12.217240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:12.217312 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:12.217568 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:12.217569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:12.217570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:12.217732 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:12.217841 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:12.218001 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 03:42:12.218086 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260806 03:42:12.218787 31975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59872
I20260806 03:42:12.219739 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 32053
I20260806 03:42:12.234474 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 32197
I20260806 03:42:12.244624 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 32349
I20260806 03:42:12.253744 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 31943
2026-08-06T03:42:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2387 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:12 dist-test-slave-10pc krb5kdc[32575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:12 dist-test-slave-10pc krb5kdc[32575](info): set up 2 sockets
Aug 06 03:42:12 dist-test-slave-10pc krb5kdc[32575](info): commencing operation
krb5kdc: starting...
W20260806 03:42:13.309317 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:13.434940 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:39503
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:13.518215 32588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:13.518437 32588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:13.518460 32588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:13.519555 32588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:13.519591 32588 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 03:42:13.519604 32588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:13.519618 32588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:13.519629 32588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:13.519641 32588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:13.521065 32588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:39503
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:13.521271 32588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:13.521448 32588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:13.526324 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:13.526319 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:13.526358 32593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:13.526714 32588 server_base.cc:1034] running on GCE node
I20260806 03:42:13.526983 32588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:13.527272 32588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:13.528497 32588 hybrid_clock.cc:648] HybridClock initialized: now 1785987733528446 us; error 41 us; skew 500 ppm
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:13.530978 32588 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:13.531446 32588 webserver.cc:460] Webserver started at http://127.30.24.62:43877/ using document root <none> and password file <none>
I20260806 03:42:13.531674 32588 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:13.531752 32588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:13.531903 32588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:13.533028 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "78c61a95768d471f8f68f3eb706a8bc3"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "decc1bacdbf593e0f0b320edf7fbf772"
server_key_iv: "a76753f3637e73c95aea4a35619cdf50"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.533411 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "78c61a95768d471f8f68f3eb706a8bc3"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "decc1bacdbf593e0f0b320edf7fbf772"
server_key_iv: "a76753f3637e73c95aea4a35619cdf50"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.534824 32588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:13.535622 32603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:13.535838 32588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:13.535949 32588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "78c61a95768d471f8f68f3eb706a8bc3"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "decc1bacdbf593e0f0b320edf7fbf772"
server_key_iv: "a76753f3637e73c95aea4a35619cdf50"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.536027 32588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:13.543572 32588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:13.545898 32588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:13.546060 32588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:13.550848 32588 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:39503
I20260806 03:42:13.550909 32666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:39503 every 8 connection(s)
I20260806 03:42:13.551220 32588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:13.551823 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:13.554174 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Bootstrap starting.
I20260806 03:42:13.554781 32667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:13.555099 32667 log.cc:824] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:13.555785 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: No bootstrap required, opened a new log
I20260806 03:42:13.557998 32667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c61a95768d471f8f68f3eb706a8bc3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39503 } }
I20260806 03:42:13.558176 32667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:13.558228 32667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c61a95768d471f8f68f3eb706a8bc3, State: Initialized, Role: FOLLOWER
I20260806 03:42:13.558386 32667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c61a95768d471f8f68f3eb706a8bc3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39503 } }
I20260806 03:42:13.558470 32667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:13.558514 32667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:13.558564 32667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:13.559108 32667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c61a95768d471f8f68f3eb706a8bc3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39503 } }
I20260806 03:42:13.559230 32667 leader_election.cc:302] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c61a95768d471f8f68f3eb706a8bc3; no voters:
I20260806 03:42:13.559414 32667 leader_election.cc:288] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:13.559456 32670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:13.559672 32667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:13.559675 32670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 78c61a95768d471f8f68f3eb706a8bc3, State: Running, Role: LEADER
I20260806 03:42:13.559800 32670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c61a95768d471f8f68f3eb706a8bc3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39503 } }
I20260806 03:42:13.559960 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 32588
I20260806 03:42:13.560060 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:13.560251 30816 external_mini_cluster.cc:1428] Setting key f4e63186f1dfb9cada990ac7ddd1dd58
I20260806 03:42:13.560534 32671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78c61a95768d471f8f68f3eb706a8bc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c61a95768d471f8f68f3eb706a8bc3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39503 } } }
I20260806 03:42:13.560635 32671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:13.560839 32680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:13.561158 32672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c61a95768d471f8f68f3eb706a8bc3. Latest consensus state: current_term: 1 leader_uuid: "78c61a95768d471f8f68f3eb706a8bc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c61a95768d471f8f68f3eb706a8bc3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39503 } } }
I20260806 03:42:13.561231 32672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:13.561431 32680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:13.562984 32680 catalog_manager.cc:1347] Generated new cluster ID: b25a61010ed7402bab697fb1d766ab15
I20260806 03:42:13.563038 32680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:13.572120 32680 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:13.572808 32680 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:13.572831 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.560784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.30.24.62:39503)
0806 03:42:13.561553 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:13.561562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:13.561594 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:42:13.561783 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:13.561791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:13.561847 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:13.561988 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:13.562734 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.564144 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.565319 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.565917 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.566014 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.568874 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:13.568880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:13.568880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:13.568893 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:13.570956 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.571362 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.571365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.571366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.571506 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.571732 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.571733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.571733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.572076 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:13.572295 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:13.572446 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 03:42:13.572608 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":74,"threads_started":1}
I20260806 03:42:13.576021 32680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Generated new TSK 0
I20260806 03:42:13.576205 32680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:13.607457 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:39503
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:13.697608 32694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:13.697855 32694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:13.697881 32694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:13.699045 32694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:13.699090 32694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:13.699111 32694 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 03:42:13.699123 32694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:13.699141 32694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:13.700727 32694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:39503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:13.700963 32694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:13.701231 32694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:13.706527 32699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:13.706663 32694 server_base.cc:1034] running on GCE node
W20260806 03:42:13.706499 32702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:13.706507 32700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:13.707118 32694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:13.707413 32694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:13.708537 32694 hybrid_clock.cc:648] HybridClock initialized: now 1785987733708514 us; error 30 us; skew 500 ppm
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:13.711175 32694 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:13.711666 32694 webserver.cc:460] Webserver started at http://127.30.24.1:45699/ using document root <none> and password file <none>
I20260806 03:42:13.711925 32694 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:13.711972 32694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:13.712142 32694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:13.713307 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "e253914ad79b48bb804cf133415f9fa6"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "2962948fd0cd7e462544ff82a70d0fff"
server_key_iv: "1f300fc7e941f98d6ca12adb3c902863"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.713680 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "e253914ad79b48bb804cf133415f9fa6"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "2962948fd0cd7e462544ff82a70d0fff"
server_key_iv: "1f300fc7e941f98d6ca12adb3c902863"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.715051 32694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:13.715801 32709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:13.715981 32694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:13.716084 32694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "e253914ad79b48bb804cf133415f9fa6"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "2962948fd0cd7e462544ff82a70d0fff"
server_key_iv: "1f300fc7e941f98d6ca12adb3c902863"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.716168 32694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:13.726450 32694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:13.728905 32694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:13.729077 32694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:13.729673 32694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:13.729733 32694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:13.729768 32694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:13.729797 32694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:13.737460 32694 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:44421
I20260806 03:42:13.737635 357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:44421 every 8 connection(s)
I20260806 03:42:13.737905 32694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:13.739619 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.730164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:59507 (local address 127.30.24.62:39503)
0806 03:42:13.730239 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:13.730242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:13.730434 (+ 192us) server_negotiation.cc:407] Connection header received
0806 03:42:13.730923 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:13.730924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:13.730943 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:13.731083 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:13.731755 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.732299 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.733380 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.733649 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.733743 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.736388 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:13.736390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:13.736390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:13.736398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:13.738033 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.738573 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.738574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.738574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.738645 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.738913 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.738914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.738914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.739039 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:13.739131 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:13.739430 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 03:42:13.739530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:42:13.739804 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.730124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.30.24.1:59507)
0806 03:42:13.730286 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:13.730327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:13.730819 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:13.731171 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:13.731174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:13.731408 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:13.731612 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.731616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.732399 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.732400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.733267 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.733270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.733757 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.733758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.733920 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.734415 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:13.734423 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:13.736287 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:13.738166 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.738174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.738177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.738473 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.738718 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.738719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.738719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.738841 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.739122 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:13.739130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:13.739276 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 03:42:13.739389 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 03:42:13.739552 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260806 03:42:13.740375 358 heartbeater.cc:347] Connected to a master server at 127.30.24.62:39503
I20260806 03:42:13.740483 358 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:13.740695 358 heartbeater.cc:511] Master 127.30.24.62:39503 requested a full tablet report, sending...
I20260806 03:42:13.741255 32621 ts_manager.cc:194] Registered new tserver with Master: e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421)
I20260806 03:42:13.742153 32621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:59507
I20260806 03:42:13.743666 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 32694
I20260806 03:42:13.743782 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:13.744001 30816 external_mini_cluster.cc:1428] Setting key 0348bea5fae7546c0f6ed5a88d2725d5
I20260806 03:42:13.747391 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.741487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.30.24.1:58527)
0806 03:42:13.741632 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:13.741643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:13.741711 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:13.741843 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:13.741843 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:13.741997 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:13.742047 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.742048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.742846 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.742847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.743407 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.743409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.743751 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.743753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.743844 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.744317 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:13.744322 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:13.744684 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:13.746320 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.746321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.746322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.746499 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.746716 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.746716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.746717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.746786 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.747037 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:13.747038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:13.747087 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 03:42:13.747179 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 03:42:13.747292 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260806 03:42:13.747385 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.741549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:58527 (local address 127.30.24.62:39503)
0806 03:42:13.741589 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:13.741591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:13.741710 (+ 119us) server_negotiation.cc:407] Connection header received
0806 03:42:13.741773 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:13.741773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:13.741783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:13.741839 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:13.742111 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.742735 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.743497 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.743676 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.743725 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.744788 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:13.744789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:13.744790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:13.744796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:13.746214 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.746575 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.746577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.746577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.746640 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.746857 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.746858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.746858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.746970 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:13.747045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:13.747205 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 03:42:13.747292 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:13.771206 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:39503
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:13.858860 368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:13.859110 368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:13.859140 368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:13.860649 368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:13.860700 368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:13.860724 368 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 03:42:13.860738 368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:13.860798 368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:13.862516 368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:39503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:13.862816 368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:13.863095 368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:13.868775 376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:13.868769 373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:13.868768 374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:13.868804 368 server_base.cc:1034] running on GCE node
I20260806 03:42:13.869269 368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:13.869525 368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:13.870733 368 hybrid_clock.cc:648] HybridClock initialized: now 1785987733870726 us; error 37 us; skew 500 ppm
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:13.873368 368 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:13.873875 368 webserver.cc:460] Webserver started at http://127.30.24.2:37347/ using document root <none> and password file <none>
I20260806 03:42:13.874126 368 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:13.874198 368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:13.874361 368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:13.875469 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "ce7c5bc147af492e9eaeea1117c3de94"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "bb4d1cdf293996484f03177076a709cf"
server_key_iv: "ba7edb70f1b84fc324698019dd1171be"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.875885 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "ce7c5bc147af492e9eaeea1117c3de94"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "bb4d1cdf293996484f03177076a709cf"
server_key_iv: "ba7edb70f1b84fc324698019dd1171be"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.877322 368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:13.878079 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:13.878253 368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:13.878362 368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "ce7c5bc147af492e9eaeea1117c3de94"
format_stamp: "Formatted at 2026-08-06 03:42:13 on dist-test-slave-10pc"
server_key: "bb4d1cdf293996484f03177076a709cf"
server_key_iv: "ba7edb70f1b84fc324698019dd1171be"
server_key_version: "encryptionkey@0"
I20260806 03:42:13.878438 368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:13.894110 368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:13.896004 368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:13.896186 368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:13.896746 368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:13.896813 368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:13.896852 368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:13.896893 368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 03:42:13 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:13.904345 368 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:36629
I20260806 03:42:13.904486 502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:36629 every 8 connection(s)
I20260806 03:42:13.904778 368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:13.907048 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 368
I20260806 03:42:13.907162 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:13.907393 30816 external_mini_cluster.cc:1428] Setting key 916736f50313bc6265293d5a5c8d23e5
I20260806 03:42:13.908442 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.897289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:40355 (local address 127.30.24.62:39503)
0806 03:42:13.897355 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:13.897357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:13.897554 (+ 197us) server_negotiation.cc:407] Connection header received
0806 03:42:13.898087 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:13.898089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:13.898107 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:13.898188 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:13.899380 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.900235 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.901549 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.901799 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.901895 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.904938 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:13.904940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:13.904940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:13.904953 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:13.906624 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.907256 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.907257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.907258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.907340 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.907685 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.907686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.907686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.907842 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:13.907931 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:13.908260 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 03:42:13.908349 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 03:42:13.908671 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.897274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.30.24.2:40355)
0806 03:42:13.897441 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:13.897479 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:13.897963 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:13.898364 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:13.898371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:13.898817 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:13.899203 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.899210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.900354 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.900355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.901419 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.901422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.901918 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.901919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.902074 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.902725 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:13.902737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:13.904830 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:13.906741 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.906750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.906752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.907146 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.907418 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.907419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.907420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.907591 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.907922 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:13.907929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:13.908103 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 03:42:13.908223 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 03:42:13.908391 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260806 03:42:13.909181 503 heartbeater.cc:347] Connected to a master server at 127.30.24.62:39503
I20260806 03:42:13.909297 503 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:13.909510 503 heartbeater.cc:511] Master 127.30.24.62:39503 requested a full tablet report, sending...
I20260806 03:42:13.909879 32620 ts_manager.cc:194] Registered new tserver with Master: ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629)
I20260806 03:42:13.910432 32620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:40355
I20260806 03:42:13.916637 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.910442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48093 (local address 127.30.24.62:39503)
0806 03:42:13.910484 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:13.910486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:13.910595 (+ 109us) server_negotiation.cc:407] Connection header received
0806 03:42:13.910660 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:13.910661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:13.910672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:13.910735 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:13.911105 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.911634 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.912487 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.912712 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.912794 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.913858 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:13.913859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:13.913859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:13.913867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:13.915308 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.915692 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.915693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.915694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.915761 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:13.915987 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:13.915988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:13.915988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:13.916156 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:13.916241 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:13.916425 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 03:42:13.916537 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:13.916647 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:13.910367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.30.24.2:48093)
0806 03:42:13.910530 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:13.910541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:13.910606 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:13.910749 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:13.910750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:13.910948 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:13.911012 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.911013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.911742 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.911743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.912370 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:13.912372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:13.912794 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:13.912795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:13.912886 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:13.913418 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:13.913424 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:13.913736 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:13.915402 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.915403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.915403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.915598 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.915835 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:13.915836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:13.915837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:13.915907 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:13.916246 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:13.916247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:13.916299 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 03:42:13.916395 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 03:42:13.916535 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:13.935969 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:39503
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:14.019903 512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:14.020110 512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:14.020135 512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:14.021332 512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:14.021374 512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:14.021394 512 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 03:42:14.021405 512 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:14.021422 512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:14.022811 512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:39503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:14.022989 512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:14.023164 512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:14.028313 517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:14.028352 512 server_base.cc:1034] running on GCE node
W20260806 03:42:14.028307 520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:14.028309 518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:14.028666 512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:14.028875 512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:14.030012 512 hybrid_clock.cc:648] HybridClock initialized: now 1785987734029991 us; error 39 us; skew 500 ppm
Aug 06 03:42:14 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987734, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:14.032622 512 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:14.033053 512 webserver.cc:460] Webserver started at http://127.30.24.3:45383/ using document root <none> and password file <none>
I20260806 03:42:14.033293 512 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:14.033361 512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:14.033519 512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:14.034576 512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "ab37a34c5fd54779b4864053681605bc"
format_stamp: "Formatted at 2026-08-06 03:42:14 on dist-test-slave-10pc"
server_key: "892f613ac5677858b7705d0d546b3751"
server_key_iv: "1581e3027104eb67ada5138a89453c56"
server_key_version: "encryptionkey@0"
I20260806 03:42:14.034965 512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "ab37a34c5fd54779b4864053681605bc"
format_stamp: "Formatted at 2026-08-06 03:42:14 on dist-test-slave-10pc"
server_key: "892f613ac5677858b7705d0d546b3751"
server_key_iv: "1581e3027104eb67ada5138a89453c56"
server_key_version: "encryptionkey@0"
I20260806 03:42:14.036360 512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:42:14.037140 527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:14.037329 512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:14.037441 512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "ab37a34c5fd54779b4864053681605bc"
format_stamp: "Formatted at 2026-08-06 03:42:14 on dist-test-slave-10pc"
server_key: "892f613ac5677858b7705d0d546b3751"
server_key_iv: "1581e3027104eb67ada5138a89453c56"
server_key_version: "encryptionkey@0"
I20260806 03:42:14.037518 512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:14.046430 512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:14.048429 512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:14.048620 512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:14.049150 512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:14.049192 512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:14.049214 512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:14.049227 512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 03:42:14 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987734, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:14.056787 512 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:40771
I20260806 03:42:14.057039 646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:40771 every 8 connection(s)
I20260806 03:42:14.057358 512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:14.059566 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.049595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52617 (local address 127.30.24.62:39503)
0806 03:42:14.049666 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.049670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.049854 (+ 184us) server_negotiation.cc:407] Connection header received
0806 03:42:14.050357 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.050358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.050371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.050463 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:14.051139 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.051954 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.053706 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.053996 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.054096 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.056444 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:14.056445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:14.056446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:14.056477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:14.058244 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.058613 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.058614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.058615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.058667 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.058941 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.058941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.058942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.059050 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:14.059100 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.059397 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.059478 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:42:14.059775 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.049555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.30.24.3:52617)
0806 03:42:14.049713 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.049754 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.050252 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.050521 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.050525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.050822 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:14.051034 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.051040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.052051 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.052052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.053603 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.053608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.054121 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.054123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.054247 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.054799 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:14.054808 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:14.056342 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:14.058317 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:14.058322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:14.058324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:14.058546 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:14.058747 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:14.058748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:14.058749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:14.058861 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:14.059124 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:14.059130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:14.059251 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.059339 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.059529 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260806 03:42:14.060410 647 heartbeater.cc:347] Connected to a master server at 127.30.24.62:39503
I20260806 03:42:14.060554 647 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:14.060770 647 heartbeater.cc:511] Master 127.30.24.62:39503 requested a full tablet report, sending...
I20260806 03:42:14.061391 32621 ts_manager.cc:194] Registered new tserver with Master: ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771)
I20260806 03:42:14.061457 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 512
I20260806 03:42:14.061539 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:14.061728 30816 external_mini_cluster.cc:1428] Setting key a3054b10ef4d52729d5a77277e411d7b
I20260806 03:42:14.061923 32621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:52617
I20260806 03:42:14.062947 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:42:14.068194 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.062043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:38727 (local address 127.30.24.62:39503)
0806 03:42:14.062097 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.062099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.062109 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:14.062141 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.062142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.062151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.062204 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:14.062649 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.063283 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.063989 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.064196 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.064268 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.065459 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:14.065460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:14.065461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:14.065469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:14.066870 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.067272 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.067274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.067274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.067398 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.067614 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.067615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.067616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.067778 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:14.067860 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.068019 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.068100 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:42:14.068200 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.061565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.30.24.3:38727)
0806 03:42:14.061931 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.061944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.061999 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.062224 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.062225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.062428 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:14.062578 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.062580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.063376 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.063377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.063898 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.063900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.064282 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.064283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.064366 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.064871 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:14.064886 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:14.065382 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:14.066991 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:14.066992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:14.066993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:14.067177 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:14.067478 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:14.067479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:14.067480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:14.067547 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:14.067853 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:14.067854 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:14.067904 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.067991 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.068102 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":84,"threads_started":1}
Aug 06 03:42:14 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 03:42:14 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:14.088416 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.081140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.30.24.62:39503)
0806 03:42:14.081226 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.081228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.081259 (+ 31us) server_negotiation.cc:407] Connection header received
0806 03:42:14.081291 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.081292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.081304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.081364 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:14.081974 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.082513 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.083271 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.083450 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.083498 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.086066 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:14.086068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:14.086068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:14.086073 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:14.087380 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.087733 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.087734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.087734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.087793 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.087963 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.087963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.087964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.088062 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:14.088135 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.088253 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.088313 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 03:42:14.090206 32621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:14.091812 32621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:14.098748 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.094200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44421 (local address 127.0.0.1:33138)
0806 03:42:14.094553 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.094563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.094623 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.094989 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.094990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.094992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.095029 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.095030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.096328 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.096329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.097681 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.097685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.098418 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.098419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.098509 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.098523 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.098596 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.098654 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":82,"thread_start_us":115,"threads_started":1}
I20260806 03:42:14.098853 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.094193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.0.0.1:58964)
0806 03:42:14.094398 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.094438 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.094505 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.095333 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.095334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.095336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.095384 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.095386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.096274 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.096275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.097511 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.097514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.098544 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.098545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.098620 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.098624 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.098674 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.098720 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":107,"thread_start_us":87,"threads_started":1}
I20260806 03:42:14.099004 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.094485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.30.24.2:36629)
0806 03:42:14.094691 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.094696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.094706 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:14.094750 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.094752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.094765 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.094873 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.095105 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.096254 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.097791 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.098514 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.098586 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.098641 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.098725 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.098765 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260806 03:42:14.099031 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.094202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36629 (local address 127.0.0.1:35214)
0806 03:42:14.094608 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.094617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.094656 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.094894 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.094897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.094900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.094969 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.094971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.096496 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.096497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.097716 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.097718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.098603 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.098604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.098658 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.098661 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.098714 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.098760 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260806 03:42:14.099378 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.094402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.30.24.1:44421)
0806 03:42:14.094730 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.094735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.094761 (+ 26us) server_negotiation.cc:407] Connection header received
0806 03:42:14.094799 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.094800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.094835 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.094917 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.095093 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.096264 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.097813 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.098320 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.098415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.098484 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.099185 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.099232 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":70,"threads_started":1}
I20260806 03:42:14.099403 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.094370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.30.24.3:40771)
0806 03:42:14.095173 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.095177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.095186 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:14.095227 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.095229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.095243 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.095334 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.095460 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.096170 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.097617 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.098368 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.098443 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.098503 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.098796 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.099011 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":86,"threads_started":1}
I20260806 03:42:14.099794 435 tablet_service.cc:1467] Processing CreateTablet for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 (DEFAULT_TABLE table=test-table [id=35788b9ad72d41e9aee34be601c504f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:14.100059 32760 tablet_service.cc:1467] Processing CreateTablet for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 (DEFAULT_TABLE table=test-table [id=35788b9ad72d41e9aee34be601c504f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:14.100188 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.100395 32760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.104149 672 tablet_bootstrap.cc:492] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Bootstrap starting.
I20260806 03:42:14.104375 671 tablet_bootstrap.cc:492] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Bootstrap starting.
I20260806 03:42:14.104537 578 tablet_service.cc:1467] Processing CreateTablet for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 (DEFAULT_TABLE table=test-table [id=35788b9ad72d41e9aee34be601c504f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:14.104871 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.105135 671 tablet_bootstrap.cc:654] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.105387 672 tablet_bootstrap.cc:654] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.105491 671 log.cc:824] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:14.105744 672 log.cc:824] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:14.106266 671 tablet_bootstrap.cc:492] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: No bootstrap required, opened a new log
I20260806 03:42:14.106323 671 ts_tablet_manager.cc:1397] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:14.107695 672 tablet_bootstrap.cc:492] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: No bootstrap required, opened a new log
I20260806 03:42:14.107766 672 ts_tablet_manager.cc:1397] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 03:42:14.108026 675 tablet_bootstrap.cc:492] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Bootstrap starting.
I20260806 03:42:14.108196 671 raft_consensus.cc:348] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.108371 671 raft_consensus.cc:374] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.108407 671 raft_consensus.cc:729] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e253914ad79b48bb804cf133415f9fa6, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.108543 671 consensus_queue.cc:260] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.108814 671 ts_tablet_manager.cc:1428] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 03:42:14.108970 358 heartbeater.cc:503] Master 127.30.24.62:39503 was elected leader, sending a full tablet report...
I20260806 03:42:14.109090 675 tablet_bootstrap.cc:654] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.109448 675 log.cc:824] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:14.109915 672 raft_consensus.cc:348] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.110108 672 raft_consensus.cc:374] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.110157 672 raft_consensus.cc:729] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.110281 672 consensus_queue.cc:260] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.110361 675 tablet_bootstrap.cc:492] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: No bootstrap required, opened a new log
I20260806 03:42:14.110446 675 ts_tablet_manager.cc:1397] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 03:42:14.110550 672 ts_tablet_manager.cc:1428] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:14.110581 503 heartbeater.cc:503] Master 127.30.24.62:39503 was elected leader, sending a full tablet report...
I20260806 03:42:14.112344 675 raft_consensus.cc:348] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.112520 675 raft_consensus.cc:374] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.112569 675 raft_consensus.cc:729] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.112682 675 consensus_queue.cc:260] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.112948 675 ts_tablet_manager.cc:1428] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:42:14.112993 647 heartbeater.cc:503] Master 127.30.24.62:39503 was elected leader, sending a full tablet report...
I20260806 03:42:14.142609 678 raft_consensus.cc:482] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:14.142738 678 raft_consensus.cc:504] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.143136 678 leader_election.cc:288] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771), e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421)
I20260806 03:42:14.146915 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.143449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:33525 (local address 127.30.24.1:44421)
0806 03:42:14.143492 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.143494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.143645 (+ 151us) server_negotiation.cc:407] Connection header received
0806 03:42:14.143718 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.143719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.143731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.143794 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.143968 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.144465 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.145981 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.146534 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.146614 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.146625 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.146802 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.146847 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:42:14.146915 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.143352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:53789 (local address 127.30.24.3:40771)
0806 03:42:14.143436 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.143438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.143522 (+ 84us) server_negotiation.cc:407] Connection header received
0806 03:42:14.143557 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.143559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.143573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.143643 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.143857 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.144442 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.145983 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.146543 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.146614 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.146625 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.146802 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.146842 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:42:14.146944 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.143288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.30.24.2:53789)
0806 03:42:14.143409 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.143434 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.143490 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.143703 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.143704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.143706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.143782 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.143784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.144558 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.144559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.145890 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.145893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.146631 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.146632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.146715 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.146722 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.146781 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.146828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":112}
I20260806 03:42:14.146955 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.143293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44421 (local address 127.30.24.2:33525)
0806 03:42:14.143569 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.143580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.143638 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.143809 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.143810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.143811 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.143900 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.143902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.144600 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.144601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.145890 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.145893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.146632 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.146632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.146715 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.146722 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.146781 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.146828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":87,"thread_start_us":133,"threads_started":1}
I20260806 03:42:14.147405 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1d2f6c2bd6474eb75f20b8e03c5ab2" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab37a34c5fd54779b4864053681605bc" is_pre_election: true
I20260806 03:42:14.147514 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1d2f6c2bd6474eb75f20b8e03c5ab2" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6" is_pre_election: true
I20260806 03:42:14.147583 599 raft_consensus.cc:2454] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 0.
I20260806 03:42:14.147691 312 raft_consensus.cc:2454] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 0.
I20260806 03:42:14.147840 387 leader_election.cc:302] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, ce7c5bc147af492e9eaeea1117c3de94; no voters:
I20260806 03:42:14.148030 678 raft_consensus.cc:2781] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:14.148097 678 raft_consensus.cc:482] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:14.148114 678 raft_consensus.cc:3037] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.148842 678 raft_consensus.cc:504] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.148990 678 leader_election.cc:288] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 election: Requested vote from peers ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771), e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421)
I20260806 03:42:14.149173 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1d2f6c2bd6474eb75f20b8e03c5ab2" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab37a34c5fd54779b4864053681605bc"
I20260806 03:42:14.149187 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1d2f6c2bd6474eb75f20b8e03c5ab2" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6"
I20260806 03:42:14.149263 599 raft_consensus.cc:3037] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.149276 312 raft_consensus.cc:3037] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.149986 599 raft_consensus.cc:2454] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 1.
I20260806 03:42:14.150043 312 raft_consensus.cc:2454] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 1.
I20260806 03:42:14.150192 387 leader_election.cc:302] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, ce7c5bc147af492e9eaeea1117c3de94; no voters:
I20260806 03:42:14.150310 678 raft_consensus.cc:2781] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:14.150493 678 raft_consensus.cc:686] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 LEADER]: Becoming Leader. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Running, Role: LEADER
I20260806 03:42:14.150604 678 consensus_queue.cc:237] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } }
I20260806 03:42:14.151506 32621 catalog_manager.cc:5495] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2). New cstate: current_term: 1 leader_uuid: "ce7c5bc147af492e9eaeea1117c3de94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } health_report { overall_health: HEALTHY } } }
W20260806 03:42:14.155541 504 tablet.cc:2369] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 03:42:14 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:14.192836 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.185485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.30.24.62:39503)
0806 03:42:14.185586 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.185588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.185759 (+ 171us) server_negotiation.cc:407] Connection header received
0806 03:42:14.185792 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.185793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.185804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.185850 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:14.186375 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.186860 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.187593 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.187777 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.187830 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.190290 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:14.190291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:14.190291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:14.190328 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:14.191569 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.191969 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.191969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.191970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.192027 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:14.192235 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:14.192235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:14.192236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:14.192352 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:14.192422 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.192616 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.192705 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 03:42:14.192927 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.185424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:39503 (local address 127.0.0.1:59602)
0806 03:42:14.185630 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.185641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.185710 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.185911 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.185912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.186244 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:14.186296 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.186299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.186931 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.186932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.187503 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.187509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.187845 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.187846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.187891 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.188551 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:14.188556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:14.190178 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:14.191666 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:14.191674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:14.191676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:14.191880 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:14.192102 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:14.192103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:14.192103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:14.192167 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:14.192425 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:14.192434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:14.192504 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.192592 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.192789 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":88,"thread_start_us":116,"threads_started":1}
I20260806 03:42:14.194979 32621 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59602:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 03:42:14.195168 32621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:14.196882 578 tablet_service.cc:1467] Processing CreateTablet for tablet 43985c09c2694d70959a6654d39dd137 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f5a77f8446648e9b87f5c51850f76ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:14.196878 435 tablet_service.cc:1467] Processing CreateTablet for tablet 43985c09c2694d70959a6654d39dd137 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f5a77f8446648e9b87f5c51850f76ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:14.196878 32760 tablet_service.cc:1467] Processing CreateTablet for tablet 43985c09c2694d70959a6654d39dd137 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f5a77f8446648e9b87f5c51850f76ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:14.197072 32760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43985c09c2694d70959a6654d39dd137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.197072 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43985c09c2694d70959a6654d39dd137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.197074 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43985c09c2694d70959a6654d39dd137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.198334 675 tablet_bootstrap.cc:492] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: Bootstrap starting.
I20260806 03:42:14.198623 672 tablet_bootstrap.cc:492] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: Bootstrap starting.
I20260806 03:42:14.198972 675 tablet_bootstrap.cc:654] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.199071 672 tablet_bootstrap.cc:654] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.199425 671 tablet_bootstrap.cc:492] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: Bootstrap starting.
I20260806 03:42:14.199805 672 tablet_bootstrap.cc:492] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: No bootstrap required, opened a new log
I20260806 03:42:14.199857 672 ts_tablet_manager.cc:1397] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:14.200016 672 raft_consensus.cc:348] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.200109 672 raft_consensus.cc:374] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.200119 671 tablet_bootstrap.cc:654] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.200133 672 raft_consensus.cc:729] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.200191 672 consensus_queue.cc:260] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.200300 672 ts_tablet_manager.cc:1428] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:14.200371 678 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.200511 678 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:14.200846 671 tablet_bootstrap.cc:492] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: No bootstrap required, opened a new log
I20260806 03:42:14.200901 671 ts_tablet_manager.cc:1397] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:14.200958 675 tablet_bootstrap.cc:492] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: No bootstrap required, opened a new log
I20260806 03:42:14.201021 675 ts_tablet_manager.cc:1397] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 03:42:14.201059 671 raft_consensus.cc:348] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.201129 671 raft_consensus.cc:374] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.201149 671 raft_consensus.cc:729] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e253914ad79b48bb804cf133415f9fa6, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.201155 675 raft_consensus.cc:348] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.201218 675 raft_consensus.cc:374] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.201241 675 raft_consensus.cc:729] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.201201 671 consensus_queue.cc:260] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.201313 671 ts_tablet_manager.cc:1428] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:14.201282 675 consensus_queue.cc:260] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.201404 675 ts_tablet_manager.cc:1428] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:14.201392 676 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.201514 676 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:14.201470 679 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.201568 679 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 03:42:14.238843 359 tablet.cc:2369] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:42:14.308672 648 tablet.cc:2369] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:14.333985 679 raft_consensus.cc:482] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:14.334102 679 raft_consensus.cc:504] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.334533 679 leader_election.cc:288] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421), ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629)
I20260806 03:42:14.337514 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.334811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:49349 (local address 127.30.24.1:44421)
0806 03:42:14.334862 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.334864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.334928 (+ 64us) server_negotiation.cc:407] Connection header received
0806 03:42:14.334997 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.334998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.335008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.335064 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.335281 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.335780 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.336798 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.337158 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.337245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.337256 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.337402 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.337442 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:42:14.337522 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.334749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44421 (local address 127.30.24.3:49349)
0806 03:42:14.334858 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.334871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.334934 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.335129 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.335130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.335133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.335202 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.335204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.335874 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.335875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.336701 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.336704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.337245 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.337246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.337324 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.337327 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.337396 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.337438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 03:42:14.337620 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.334849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:54913 (local address 127.30.24.2:36629)
0806 03:42:14.334918 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.334920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.335037 (+ 117us) server_negotiation.cc:407] Connection header received
0806 03:42:14.335173 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.335176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.335191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.335254 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:14.335419 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.335893 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.336842 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.337246 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.337333 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.337344 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.337510 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.337547 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:42:14.337630 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.334765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36629 (local address 127.30.24.3:54913)
0806 03:42:14.334948 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.334958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.335051 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.335260 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.335261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.335262 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:14.335339 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.335341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.335982 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.335983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.336740 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.336742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.337357 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.337358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.337434 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.337438 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.337504 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.337543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260806 03:42:14.337797 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43985c09c2694d70959a6654d39dd137" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6" is_pre_election: true
I20260806 03:42:14.337917 312 raft_consensus.cc:2454] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 0.
I20260806 03:42:14.337982 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43985c09c2694d70959a6654d39dd137" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7c5bc147af492e9eaeea1117c3de94" is_pre_election: true
I20260806 03:42:14.338117 455 raft_consensus.cc:2454] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 0.
I20260806 03:42:14.338158 529 leader_election.cc:302] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, e253914ad79b48bb804cf133415f9fa6; no voters:
I20260806 03:42:14.338402 679 raft_consensus.cc:2781] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:14.338442 679 raft_consensus.cc:482] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:14.338457 679 raft_consensus.cc:3037] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.339305 679 raft_consensus.cc:504] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.339464 679 leader_election.cc:288] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 election: Requested vote from peers e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421), ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629)
I20260806 03:42:14.339670 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43985c09c2694d70959a6654d39dd137" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6"
I20260806 03:42:14.339763 312 raft_consensus.cc:3037] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.339744 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43985c09c2694d70959a6654d39dd137" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7c5bc147af492e9eaeea1117c3de94"
I20260806 03:42:14.339823 455 raft_consensus.cc:3037] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.340699 312 raft_consensus.cc:2454] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 1.
I20260806 03:42:14.340726 455 raft_consensus.cc:2454] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 1.
I20260806 03:42:14.340883 531 leader_election.cc:302] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, ce7c5bc147af492e9eaeea1117c3de94; no voters:
I20260806 03:42:14.340973 679 raft_consensus.cc:2781] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:14.341197 679 raft_consensus.cc:686] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [term 1 LEADER]: Becoming Leader. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Running, Role: LEADER
I20260806 03:42:14.341318 679 consensus_queue.cc:237] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.341786 694 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: TxnStatusTablet state changed. Reason: New leader ab37a34c5fd54779b4864053681605bc. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.341904 694 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:14.342113 696 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 03:42:14.342254 696 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 03:42:14.342222 32619 catalog_manager.cc:5495] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ab37a34c5fd54779b4864053681605bc (127.30.24.3). New cstate: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:14.348099 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.345596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.30.24.3:40771)
0806 03:42:14.345663 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.345665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.345675 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:14.345751 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.345752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.345769 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.345835 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:14.345964 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.346470 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.347254 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.347464 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.347549 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.347761 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.347860 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.347949 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.347999 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.348022 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:42:14.348117 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.345466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.0.0.1:58974)
0806 03:42:14.345574 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.345606 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.345671 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.345841 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.345842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.345843 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:14.345893 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.345896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.346560 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.346561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.347162 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.347164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.347551 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.347552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.347631 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.347653 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.347930 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.347944 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.347994 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.348046 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 03:42:14.349931 312 raft_consensus.cc:1261] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Refusing update from remote peer ab37a34c5fd54779b4864053681605bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:14.349969 455 raft_consensus.cc:1261] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Refusing update from remote peer ab37a34c5fd54779b4864053681605bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:14.350294 679 consensus_queue.cc:1035] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:14.350453 694 consensus_queue.cc:1035] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:14.351532 678 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: TxnStatusTablet state changed. Reason: New leader ab37a34c5fd54779b4864053681605bc. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.351635 678 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:14.352813 676 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: TxnStatusTablet state changed. Reason: New leader ab37a34c5fd54779b4864053681605bc. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.352911 676 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:14.353226 694 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.353309 694 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:14.353346 699 mvcc.cc:205] Tried to move back new op lower bound from 7315405759894786048 to 7315405759862677504. Current Snapshot: MvccSnapshot[applied={T|T < 7315405759894786048}]
I20260806 03:42:14.353874 678 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.353976 678 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ce7c5bc147af492e9eaeea1117c3de94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:14.354936 679 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.355026 679 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P ab37a34c5fd54779b4864053681605bc: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:14.355515 676 tablet_replica.cc:439] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } } }
I20260806 03:42:14.355604 676 tablet_replica.cc:442] T 43985c09c2694d70959a6654d39dd137 P e253914ad79b48bb804cf133415f9fa6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:14.358652 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.355480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.30.24.2:36629)
0806 03:42:14.355516 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.355518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.355733 (+ 215us) server_negotiation.cc:407] Connection header received
0806 03:42:14.355818 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.355819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.355835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.355922 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:14.356149 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.356875 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.357772 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.358030 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.358094 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.358292 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.358406 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.358500 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.358523 (+ 23us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.358564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:42:14.363945 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.361455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56695 (local address 127.30.24.3:40771)
0806 03:42:14.361516 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.361519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.361544 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:42:14.361568 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.361569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.361579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.361630 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:14.361773 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.362211 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.363300 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.363472 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.363522 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.363729 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.363755 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.363793 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.363871 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.363888 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:42:14.363994 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.361386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.30.24.2:56695)
0806 03:42:14.361457 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.361467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.361510 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.361649 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.361651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.361652 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:14.361711 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.361713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.362279 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.362280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.363216 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.363219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.363543 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.363544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.363614 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.363631 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.363812 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.363814 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.363870 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.363916 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 03:42:14.366926 599 raft_consensus.cc:1261] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Refusing update from remote peer ce7c5bc147af492e9eaeea1117c3de94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:14.367118 312 raft_consensus.cc:1261] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Refusing update from remote peer ce7c5bc147af492e9eaeea1117c3de94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:14.367239 681 consensus_queue.cc:1035] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:14.367543 681 consensus_queue.cc:1035] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:14.369128 700 mvcc.cc:205] Tried to move back new op lower bound from 7315405759964676096 to 7315405759081754624. Current Snapshot: MvccSnapshot[applied={T|T < 7315405759964676096}]
I20260806 03:42:14.370925 702 mvcc.cc:205] Tried to move back new op lower bound from 7315405759964676096 to 7315405759081754624. Current Snapshot: MvccSnapshot[applied={T|T < 7315405759964676096}]
I20260806 03:42:14.377668 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.374861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:53917 (local address 127.30.24.2:36629)
0806 03:42:14.374902 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.374904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.374931 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:42:14.374985 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.374986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.374999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.375066 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:14.375163 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.375922 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.376801 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.377125 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.377198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.377416 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.377452 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.377504 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.377591 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.377615 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:14.377719 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.374801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36629 (local address 127.30.24.3:53917)
0806 03:42:14.374878 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.374893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.374940 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.375067 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.375068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.375069 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:14.375127 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.375128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.375994 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.375996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.376720 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.376722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.377238 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.377239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.377312 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.377328 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.377532 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.377535 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.377583 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.377638 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 03:42:14.388623 32620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:14.388743 32620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:14.390662 32620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Sending DeleteTablet for 3 replicas of tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2
I20260806 03:42:14.391360 32760 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:14 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33138
I20260806 03:42:14.391953 435 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:14 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:35214
I20260806 03:42:14.392241 720 tablet_replica.cc:330] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: stopping tablet replica
I20260806 03:42:14.392395 720 raft_consensus.cc:2229] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:14.392606 720 raft_consensus.cc:2258] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:14.392629 578 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:14 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:58964
I20260806 03:42:14.392925 32620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:14.393105 32620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:14.393348 721 tablet_replica.cc:330] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: stopping tablet replica
I20260806 03:42:14.393460 721 raft_consensus.cc:2229] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:14.393533 721 raft_consensus.cc:2258] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:14.393685 719 tablet_replica.cc:330] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: stopping tablet replica
I20260806 03:42:14.393793 719 raft_consensus.cc:2229] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:14.393846 719 raft_consensus.cc:2258] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:14.394312 719 ts_tablet_manager.cc:1905] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:14.394968 32760 tablet_service.cc:1467] Processing CreateTablet for tablet 39a260b0f332417e9b992dcfe35395a1 (DEFAULT_TABLE table=test-table [id=96c7f2446bdf498c8a0ef990b375ca2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:14.395088 32760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a260b0f332417e9b992dcfe35395a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.396222 719 ts_tablet_manager.cc:1918] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:14.396294 719 log.cc:1196] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/ce1d2f6c2bd6474eb75f20b8e03c5ab2
I20260806 03:42:14.396477 435 tablet_service.cc:1467] Processing CreateTablet for tablet 39a260b0f332417e9b992dcfe35395a1 (DEFAULT_TABLE table=test-table [id=96c7f2446bdf498c8a0ef990b375ca2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:14.396615 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a260b0f332417e9b992dcfe35395a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.396612 719 ts_tablet_manager.cc:1939] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P e253914ad79b48bb804cf133415f9fa6: Deleting consensus metadata
I20260806 03:42:14.397156 721 ts_tablet_manager.cc:1905] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:14.397919 671 tablet_bootstrap.cc:492] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Bootstrap starting.
I20260806 03:42:14.398322 671 tablet_bootstrap.cc:654] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.398728 672 tablet_bootstrap.cc:492] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Bootstrap starting.
I20260806 03:42:14.398900 721 ts_tablet_manager.cc:1918] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:14.398969 721 log.cc:1196] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/ce1d2f6c2bd6474eb75f20b8e03c5ab2
I20260806 03:42:14.399134 672 tablet_bootstrap.cc:654] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.399235 721 ts_tablet_manager.cc:1939] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ab37a34c5fd54779b4864053681605bc: Deleting consensus metadata
I20260806 03:42:14.399756 32605 catalog_manager.cc:4841] TS e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421): tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 (table test-table [id=35788b9ad72d41e9aee34be601c504f7]) successfully deleted
I20260806 03:42:14.399963 32606 catalog_manager.cc:4841] TS ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771): tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 (table test-table [id=35788b9ad72d41e9aee34be601c504f7]) successfully deleted
I20260806 03:42:14.400055 578 tablet_service.cc:1467] Processing CreateTablet for tablet 39a260b0f332417e9b992dcfe35395a1 (DEFAULT_TABLE table=test-table [id=96c7f2446bdf498c8a0ef990b375ca2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:14.400156 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a260b0f332417e9b992dcfe35395a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:14.400696 720 ts_tablet_manager.cc:1905] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:14.401347 675 tablet_bootstrap.cc:492] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Bootstrap starting.
I20260806 03:42:14.401787 675 tablet_bootstrap.cc:654] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:14.402472 675 tablet_bootstrap.cc:492] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: No bootstrap required, opened a new log
I20260806 03:42:14.402544 675 ts_tablet_manager.cc:1397] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:14.402689 675 raft_consensus.cc:348] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.402769 675 raft_consensus.cc:374] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.402796 675 raft_consensus.cc:729] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.402832 675 consensus_queue.cc:260] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.402916 675 ts_tablet_manager.cc:1428] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:14.403121 720 ts_tablet_manager.cc:1918] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:14.403240 720 log.cc:1196] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/ce1d2f6c2bd6474eb75f20b8e03c5ab2
I20260806 03:42:14.403584 720 ts_tablet_manager.cc:1939] T ce1d2f6c2bd6474eb75f20b8e03c5ab2 P ce7c5bc147af492e9eaeea1117c3de94: Deleting consensus metadata
I20260806 03:42:14.404043 32607 catalog_manager.cc:4841] TS ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629): tablet ce1d2f6c2bd6474eb75f20b8e03c5ab2 (table test-table [id=35788b9ad72d41e9aee34be601c504f7]) successfully deleted
I20260806 03:42:14.404876 672 tablet_bootstrap.cc:492] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: No bootstrap required, opened a new log
I20260806 03:42:14.404928 672 ts_tablet_manager.cc:1397] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 03:42:14.405088 672 raft_consensus.cc:348] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.405154 672 raft_consensus.cc:374] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.405174 672 raft_consensus.cc:729] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.405215 672 consensus_queue.cc:260] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.405292 672 ts_tablet_manager.cc:1428] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:14.405515 671 tablet_bootstrap.cc:492] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: No bootstrap required, opened a new log
I20260806 03:42:14.405558 671 ts_tablet_manager.cc:1397] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260806 03:42:14.405721 671 raft_consensus.cc:348] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.405787 671 raft_consensus.cc:374] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:14.405807 671 raft_consensus.cc:729] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e253914ad79b48bb804cf133415f9fa6, State: Initialized, Role: FOLLOWER
I20260806 03:42:14.405854 671 consensus_queue.cc:260] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.405942 671 ts_tablet_manager.cc:1428] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:14.597651 679 raft_consensus.cc:482] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:14.597759 679 raft_consensus.cc:504] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.597932 679 leader_election.cc:288] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421), ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629)
I20260806 03:42:14.598235 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a260b0f332417e9b992dcfe35395a1" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7c5bc147af492e9eaeea1117c3de94" is_pre_election: true
I20260806 03:42:14.598237 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a260b0f332417e9b992dcfe35395a1" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6" is_pre_election: true
I20260806 03:42:14.598333 455 raft_consensus.cc:2454] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 0.
I20260806 03:42:14.598361 312 raft_consensus.cc:2454] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 0.
I20260806 03:42:14.598508 531 leader_election.cc:302] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, ce7c5bc147af492e9eaeea1117c3de94; no voters:
I20260806 03:42:14.598635 679 raft_consensus.cc:2781] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:14.598681 679 raft_consensus.cc:482] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:14.598699 679 raft_consensus.cc:3037] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.599538 679 raft_consensus.cc:504] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.599680 679 leader_election.cc:288] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 election: Requested vote from peers e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421), ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629)
I20260806 03:42:14.599861 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a260b0f332417e9b992dcfe35395a1" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6"
I20260806 03:42:14.599861 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a260b0f332417e9b992dcfe35395a1" candidate_uuid: "ab37a34c5fd54779b4864053681605bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7c5bc147af492e9eaeea1117c3de94"
I20260806 03:42:14.599946 312 raft_consensus.cc:3037] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.599946 455 raft_consensus.cc:3037] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:14.600813 455 raft_consensus.cc:2454] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 1.
I20260806 03:42:14.600813 312 raft_consensus.cc:2454] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab37a34c5fd54779b4864053681605bc in term 1.
I20260806 03:42:14.601033 529 leader_election.cc:302] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, e253914ad79b48bb804cf133415f9fa6; no voters:
I20260806 03:42:14.601173 679 raft_consensus.cc:2781] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:14.601255 679 raft_consensus.cc:686] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 1 LEADER]: Becoming Leader. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Running, Role: LEADER
I20260806 03:42:14.601312 679 consensus_queue.cc:237] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:14.601886 32619 catalog_manager.cc:5495] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ab37a34c5fd54779b4864053681605bc (127.30.24.3). New cstate: current_term: 1 leader_uuid: "ab37a34c5fd54779b4864053681605bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:14.633893 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.631685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.30.24.3:40771)
0806 03:42:14.631738 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.631740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.631748 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:14.631771 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.631772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.631783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.631833 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:14.631948 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.632420 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.633191 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.633395 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.633477 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.633628 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.633665 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.633726 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.633784 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.633804 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:42:14.638352 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.636108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.30.24.3:50969)
0806 03:42:14.636213 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:14.636226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:14.636280 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:14.636424 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:14.636425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:14.636426 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:14.636485 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.636487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.637082 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.637083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.637614 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:14.637616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.637960 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.637961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:14.638015 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.638019 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.638194 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.638196 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:14.638249 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 03:42:14.638284 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 03:42:14.638360 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:14.636180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:50969 (local address 127.30.24.3:40771)
0806 03:42:14.636234 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:14.636236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:14.636263 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:42:14.636338 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:14.636339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:14.636349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:14.636420 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:14.636545 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.636996 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.637701 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:14.637876 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:14.637961 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:14.638100 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:14.638133 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:14.638193 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:14.638253 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 03:42:14.638268 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:42:14.640594 455 raft_consensus.cc:1261] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Refusing update from remote peer ab37a34c5fd54779b4864053681605bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:14.640671 312 raft_consensus.cc:1261] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Refusing update from remote peer ab37a34c5fd54779b4864053681605bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:14.640894 679 consensus_queue.cc:1035] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:14.641099 679 consensus_queue.cc:1035] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:42:14.645752 699 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 03:42:16.166080 32619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:16.166193 32619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:16.167150 32619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Sending DeleteTablet for 3 replicas of tablet 39a260b0f332417e9b992dcfe35395a1
I20260806 03:42:16.167472 435 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a260b0f332417e9b992dcfe35395a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:16 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:35214
I20260806 03:42:16.167493 578 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a260b0f332417e9b992dcfe35395a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:16 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:58964
I20260806 03:42:16.167524 32760 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a260b0f332417e9b992dcfe35395a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:16 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33138
I20260806 03:42:16.167721 750 tablet_replica.cc:330] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: stopping tablet replica
I20260806 03:42:16.167778 750 raft_consensus.cc:2229] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:16.167865 750 raft_consensus.cc:2258] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:16.168103 750 ts_tablet_manager.cc:1905] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:16.169996 751 tablet_replica.cc:330] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: stopping tablet replica
I20260806 03:42:16.170039 752 tablet_replica.cc:330] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: stopping tablet replica
I20260806 03:42:16.170092 752 raft_consensus.cc:2229] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:16.170080 751 raft_consensus.cc:2229] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:16.170131 752 raft_consensus.cc:2258] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:16.170131 751 raft_consensus.cc:2258] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:16.170133 750 ts_tablet_manager.cc:1918] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 03:42:16.170197 750 log.cc:1196] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/39a260b0f332417e9b992dcfe35395a1
I20260806 03:42:16.170506 750 ts_tablet_manager.cc:1939] T 39a260b0f332417e9b992dcfe35395a1 P ab37a34c5fd54779b4864053681605bc: Deleting consensus metadata
I20260806 03:42:16.170712 752 ts_tablet_manager.cc:1905] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:16.170848 32606 catalog_manager.cc:4841] TS ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771): tablet 39a260b0f332417e9b992dcfe35395a1 (table test-table [id=96c7f2446bdf498c8a0ef990b375ca2e]) successfully deleted
I20260806 03:42:16.171813 751 ts_tablet_manager.cc:1905] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:16.172597 752 ts_tablet_manager.cc:1918] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 03:42:16.172673 752 log.cc:1196] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/39a260b0f332417e9b992dcfe35395a1
I20260806 03:42:16.172904 752 ts_tablet_manager.cc:1939] T 39a260b0f332417e9b992dcfe35395a1 P e253914ad79b48bb804cf133415f9fa6: Deleting consensus metadata
I20260806 03:42:16.173326 32605 catalog_manager.cc:4841] TS e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421): tablet 39a260b0f332417e9b992dcfe35395a1 (table test-table [id=96c7f2446bdf498c8a0ef990b375ca2e]) successfully deleted
I20260806 03:42:16.173610 751 ts_tablet_manager.cc:1918] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 03:42:16.173673 751 log.cc:1196] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/39a260b0f332417e9b992dcfe35395a1
I20260806 03:42:16.173921 751 ts_tablet_manager.cc:1939] T 39a260b0f332417e9b992dcfe35395a1 P ce7c5bc147af492e9eaeea1117c3de94: Deleting consensus metadata
I20260806 03:42:16.174242 32607 catalog_manager.cc:4841] TS ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629): tablet 39a260b0f332417e9b992dcfe35395a1 (table test-table [id=96c7f2446bdf498c8a0ef990b375ca2e]) successfully deleted
Aug 06 03:42:16 dist-test-slave-10pc krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987736, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 03:42:16 dist-test-slave-10pc krb5kdc[32575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987736, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:16.199990 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.187084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.30.24.62:39503)
0806 03:42:16.187299 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.187301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.187311 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:16.187343 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.187343 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.187355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.187425 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:16.188025 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.188710 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.189551 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.189781 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.189845 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.197093 (+ 7248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:16.197095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:16.197095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:16.197104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:16.198876 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:16.199234 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:16.199235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:16.199236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:16.199311 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:16.199487 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:16.199487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:16.199488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:16.199611 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:16.199696 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.199787 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.199861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":114,"threads_started":1}
I20260806 03:42:16.201663 32619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:16.201853 32619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:16.203331 435 tablet_service.cc:1467] Processing CreateTablet for tablet 40fc184a05694ec2af2b800c8a691c38 (DEFAULT_TABLE table=test-table [id=90cda09a139e495caaa5f33f2c300c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:16.203482 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fc184a05694ec2af2b800c8a691c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:16.203457 578 tablet_service.cc:1467] Processing CreateTablet for tablet 40fc184a05694ec2af2b800c8a691c38 (DEFAULT_TABLE table=test-table [id=90cda09a139e495caaa5f33f2c300c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:16.203459 32760 tablet_service.cc:1467] Processing CreateTablet for tablet 40fc184a05694ec2af2b800c8a691c38 (DEFAULT_TABLE table=test-table [id=90cda09a139e495caaa5f33f2c300c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:16.203568 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fc184a05694ec2af2b800c8a691c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:16.203590 32760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fc184a05694ec2af2b800c8a691c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:16.204802 761 tablet_bootstrap.cc:492] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Bootstrap starting.
I20260806 03:42:16.205060 763 tablet_bootstrap.cc:492] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Bootstrap starting.
I20260806 03:42:16.205238 761 tablet_bootstrap.cc:654] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:16.205474 763 tablet_bootstrap.cc:654] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:16.205829 762 tablet_bootstrap.cc:492] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Bootstrap starting.
I20260806 03:42:16.205891 761 tablet_bootstrap.cc:492] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: No bootstrap required, opened a new log
I20260806 03:42:16.205931 761 ts_tablet_manager.cc:1397] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.206115 761 raft_consensus.cc:348] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.206187 763 tablet_bootstrap.cc:492] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: No bootstrap required, opened a new log
I20260806 03:42:16.206218 761 raft_consensus.cc:374] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:16.206224 762 tablet_bootstrap.cc:654] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:16.206247 761 raft_consensus.cc:729] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e253914ad79b48bb804cf133415f9fa6, State: Initialized, Role: FOLLOWER
I20260806 03:42:16.206256 763 ts_tablet_manager.cc:1397] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.206322 761 consensus_queue.cc:260] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.206382 763 raft_consensus.cc:348] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.206459 763 raft_consensus.cc:374] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:16.206488 763 raft_consensus.cc:729] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Initialized, Role: FOLLOWER
I20260806 03:42:16.206501 761 ts_tablet_manager.cc:1428] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.206575 763 consensus_queue.cc:260] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.206768 762 tablet_bootstrap.cc:492] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: No bootstrap required, opened a new log
I20260806 03:42:16.206821 762 ts_tablet_manager.cc:1397] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.206887 763 ts_tablet_manager.cc:1428] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.206934 762 raft_consensus.cc:348] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.207012 762 raft_consensus.cc:374] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:16.207036 762 raft_consensus.cc:729] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Initialized, Role: FOLLOWER
I20260806 03:42:16.207088 762 consensus_queue.cc:260] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.207217 762 ts_tablet_manager.cc:1428] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:16.369112 768 raft_consensus.cc:482] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:16.369221 768 raft_consensus.cc:504] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.369443 768 leader_election.cc:288] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421), ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771)
I20260806 03:42:16.369675 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc184a05694ec2af2b800c8a691c38" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab37a34c5fd54779b4864053681605bc" is_pre_election: true
I20260806 03:42:16.369709 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc184a05694ec2af2b800c8a691c38" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6" is_pre_election: true
I20260806 03:42:16.369804 599 raft_consensus.cc:2454] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 0.
I20260806 03:42:16.369804 312 raft_consensus.cc:2454] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 0.
I20260806 03:42:16.370002 387 leader_election.cc:302] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, ce7c5bc147af492e9eaeea1117c3de94; no voters:
I20260806 03:42:16.370141 768 raft_consensus.cc:2781] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:16.370218 768 raft_consensus.cc:482] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:16.370266 768 raft_consensus.cc:3037] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:16.371037 768 raft_consensus.cc:504] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.371204 768 leader_election.cc:288] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 election: Requested vote from peers e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421), ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771)
I20260806 03:42:16.371410 312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc184a05694ec2af2b800c8a691c38" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e253914ad79b48bb804cf133415f9fa6"
I20260806 03:42:16.371414 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc184a05694ec2af2b800c8a691c38" candidate_uuid: "ce7c5bc147af492e9eaeea1117c3de94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab37a34c5fd54779b4864053681605bc"
I20260806 03:42:16.371502 599 raft_consensus.cc:3037] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:16.371503 312 raft_consensus.cc:3037] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:16.372258 312 raft_consensus.cc:2454] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 1.
I20260806 03:42:16.372407 599 raft_consensus.cc:2454] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7c5bc147af492e9eaeea1117c3de94 in term 1.
I20260806 03:42:16.372479 385 leader_election.cc:302] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7c5bc147af492e9eaeea1117c3de94, e253914ad79b48bb804cf133415f9fa6; no voters:
I20260806 03:42:16.372581 768 raft_consensus.cc:2781] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:16.372772 768 raft_consensus.cc:686] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 LEADER]: Becoming Leader. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Running, Role: LEADER
I20260806 03:42:16.372860 768 consensus_queue.cc:237] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.373529 32619 catalog_manager.cc:5495] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2). New cstate: current_term: 1 leader_uuid: "ce7c5bc147af492e9eaeea1117c3de94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:16.393157 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.390910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.30.24.2:36629)
0806 03:42:16.391085 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.391087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.391095 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:16.391125 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.391126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.391136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.391196 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:16.391315 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.391791 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.392570 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.392781 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.392845 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.392952 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:16.392980 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:16.393014 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.393067 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.393083 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260806 03:42:16.395675 599 raft_consensus.cc:1261] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Refusing update from remote peer ce7c5bc147af492e9eaeea1117c3de94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:16.395881 312 raft_consensus.cc:1261] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Refusing update from remote peer ce7c5bc147af492e9eaeea1117c3de94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:16.395928 769 consensus_queue.cc:1035] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:16.396145 769 consensus_queue.cc:1035] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:16.407909 32621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:16.408008 32621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:16.408689 32621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Sending DeleteTablet for 3 replicas of tablet 40fc184a05694ec2af2b800c8a691c38
I20260806 03:42:16.408937 32760 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fc184a05694ec2af2b800c8a691c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:16 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33138
I20260806 03:42:16.409003 435 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fc184a05694ec2af2b800c8a691c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:16 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:35214
I20260806 03:42:16.409102 751 tablet_replica.cc:330] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: stopping tablet replica
I20260806 03:42:16.409103 752 tablet_replica.cc:330] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: stopping tablet replica
I20260806 03:42:16.409193 752 raft_consensus.cc:2229] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:16.409190 751 raft_consensus.cc:2229] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:16.409241 752 raft_consensus.cc:2258] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:16.409265 751 raft_consensus.cc:2258] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:16.409322 578 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fc184a05694ec2af2b800c8a691c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:16 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:58964
I20260806 03:42:16.409425 750 tablet_replica.cc:330] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: stopping tablet replica
I20260806 03:42:16.409487 750 raft_consensus.cc:2229] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:16.409521 750 raft_consensus.cc:2258] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:16.409600 751 ts_tablet_manager.cc:1905] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:16.409680 752 ts_tablet_manager.cc:1905] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:16.409649 32621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:16.409782 32621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:16.410977 435 tablet_service.cc:1467] Processing CreateTablet for tablet cdf7e179c006423e9f05dcba4d859988 (DEFAULT_TABLE table=test-table [id=aeb206e199f34a9f8575e8328a51bf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:16.411019 32760 tablet_service.cc:1467] Processing CreateTablet for tablet cdf7e179c006423e9f05dcba4d859988 (DEFAULT_TABLE table=test-table [id=aeb206e199f34a9f8575e8328a51bf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:16.411091 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf7e179c006423e9f05dcba4d859988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:16.411270 578 tablet_service.cc:1467] Processing CreateTablet for tablet cdf7e179c006423e9f05dcba4d859988 (DEFAULT_TABLE table=test-table [id=aeb206e199f34a9f8575e8328a51bf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:16.411347 752 ts_tablet_manager.cc:1918] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:16.411399 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf7e179c006423e9f05dcba4d859988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:16.411412 752 log.cc:1196] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/40fc184a05694ec2af2b800c8a691c38
I20260806 03:42:16.411615 750 ts_tablet_manager.cc:1905] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:16.411651 752 ts_tablet_manager.cc:1939] T 40fc184a05694ec2af2b800c8a691c38 P e253914ad79b48bb804cf133415f9fa6: Deleting consensus metadata
I20260806 03:42:16.411810 32760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf7e179c006423e9f05dcba4d859988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:16.412866 762 tablet_bootstrap.cc:492] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Bootstrap starting.
I20260806 03:42:16.413053 761 tablet_bootstrap.cc:492] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6: Bootstrap starting.
I20260806 03:42:16.413244 751 ts_tablet_manager.cc:1918] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:16.413269 750 ts_tablet_manager.cc:1918] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:16.413297 751 log.cc:1196] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/40fc184a05694ec2af2b800c8a691c38
I20260806 03:42:16.413312 762 tablet_bootstrap.cc:654] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:16.413533 751 ts_tablet_manager.cc:1939] T 40fc184a05694ec2af2b800c8a691c38 P ce7c5bc147af492e9eaeea1117c3de94: Deleting consensus metadata
I20260806 03:42:16.413313 750 log.cc:1196] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/40fc184a05694ec2af2b800c8a691c38
I20260806 03:42:16.413745 761 tablet_bootstrap.cc:654] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:16.413905 750 ts_tablet_manager.cc:1939] T 40fc184a05694ec2af2b800c8a691c38 P ab37a34c5fd54779b4864053681605bc: Deleting consensus metadata
I20260806 03:42:16.414007 32605 catalog_manager.cc:4841] TS e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421): tablet 40fc184a05694ec2af2b800c8a691c38 (table test-table [id=90cda09a139e495caaa5f33f2c300c25]) successfully deleted
I20260806 03:42:16.414127 32607 catalog_manager.cc:4841] TS ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629): tablet 40fc184a05694ec2af2b800c8a691c38 (table test-table [id=90cda09a139e495caaa5f33f2c300c25]) successfully deleted
I20260806 03:42:16.414304 762 tablet_bootstrap.cc:492] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: No bootstrap required, opened a new log
I20260806 03:42:16.414345 762 ts_tablet_manager.cc:1397] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.414351 761 tablet_bootstrap.cc:492] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6: No bootstrap required, opened a new log
I20260806 03:42:16.414384 761 ts_tablet_manager.cc:1397] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.414443 762 raft_consensus.cc:348] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.414500 762 raft_consensus.cc:374] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:16.414492 761 raft_consensus.cc:348] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.414526 762 raft_consensus.cc:729] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab37a34c5fd54779b4864053681605bc, State: Initialized, Role: FOLLOWER
I20260806 03:42:16.414542 761 raft_consensus.cc:374] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:16.414556 761 raft_consensus.cc:729] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e253914ad79b48bb804cf133415f9fa6, State: Initialized, Role: FOLLOWER
I20260806 03:42:16.414557 762 consensus_queue.cc:260] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.414616 761 consensus_queue.cc:260] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.414646 762 ts_tablet_manager.cc:1428] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:16.414709 761 ts_tablet_manager.cc:1428] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:16.415412 763 tablet_bootstrap.cc:492] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Bootstrap starting.
I20260806 03:42:16.415763 763 tablet_bootstrap.cc:654] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:16.416534 763 tablet_bootstrap.cc:492] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: No bootstrap required, opened a new log
I20260806 03:42:16.416612 763 ts_tablet_manager.cc:1397] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:16.416621 32606 catalog_manager.cc:4841] TS ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771): tablet 40fc184a05694ec2af2b800c8a691c38 (table test-table [id=90cda09a139e495caaa5f33f2c300c25]) successfully deleted
I20260806 03:42:16.416719 763 raft_consensus.cc:348] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.416788 763 raft_consensus.cc:374] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:16.416803 763 raft_consensus.cc:729] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7c5bc147af492e9eaeea1117c3de94, State: Initialized, Role: FOLLOWER
I20260806 03:42:16.416855 763 consensus_queue.cc:260] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.416951 763 ts_tablet_manager.cc:1428] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:16.464133 767 raft_consensus.cc:482] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:16.464265 767 raft_consensus.cc:504] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.464722 767 leader_election.cc:288] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629), ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771)
I20260806 03:42:16.469316 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.465111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:57695 (local address 127.30.24.2:36629)
0806 03:42:16.465201 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.465204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.465249 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:42:16.465276 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.465277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.465288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.465356 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:16.465652 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.466437 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.468180 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.468796 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.468883 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.468897 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.469175 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.469244 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 03:42:16.469362 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.464938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.30.24.1:45049)
0806 03:42:16.465157 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:16.465165 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:16.465207 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:16.465416 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:16.465418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:16.465421 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:16.465549 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.465552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.466577 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.466577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.468068 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.468070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.468988 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.468990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.469154 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.469157 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:16.469212 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 03:42:16.469242 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":117,"thread_start_us":94,"threads_started":1}
I20260806 03:42:16.469352 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.465034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:45049 (local address 127.30.24.3:40771)
0806 03:42:16.465201 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.465204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.465216 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:16.465275 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.465275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.465288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.465369 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:16.465653 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.466420 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.468180 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.468802 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.468883 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.468897 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.469228 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.469278 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260806 03:42:16.469362 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.464938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36629 (local address 127.30.24.1:57695)
0806 03:42:16.465114 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:16.465127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:16.465193 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:16.465420 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:16.465421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:16.465423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:16.465549 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.465552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.466555 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.466556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.468015 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.468019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.468989 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.468990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.469080 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.469088 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 03:42:16.469156 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 03:42:16.469217 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":105,"thread_start_us":148,"threads_started":1}
I20260806 03:42:16.469866 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf7e179c006423e9f05dcba4d859988" candidate_uuid: "e253914ad79b48bb804cf133415f9fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab37a34c5fd54779b4864053681605bc" is_pre_election: true
I20260806 03:42:16.469890 454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf7e179c006423e9f05dcba4d859988" candidate_uuid: "e253914ad79b48bb804cf133415f9fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7c5bc147af492e9eaeea1117c3de94" is_pre_election: true
I20260806 03:42:16.469980 599 raft_consensus.cc:2454] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e253914ad79b48bb804cf133415f9fa6 in term 0.
I20260806 03:42:16.469977 454 raft_consensus.cc:2454] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e253914ad79b48bb804cf133415f9fa6 in term 0.
I20260806 03:42:16.470186 32712 leader_election.cc:302] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, e253914ad79b48bb804cf133415f9fa6; no voters:
I20260806 03:42:16.470328 767 raft_consensus.cc:2781] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:16.470373 767 raft_consensus.cc:482] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:16.470393 767 raft_consensus.cc:3037] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:16.471244 767 raft_consensus.cc:504] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.471441 767 leader_election.cc:288] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [CANDIDATE]: Term 1 election: Requested vote from peers ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629), ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771)
I20260806 03:42:16.471643 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf7e179c006423e9f05dcba4d859988" candidate_uuid: "e253914ad79b48bb804cf133415f9fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab37a34c5fd54779b4864053681605bc"
I20260806 03:42:16.471681 454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf7e179c006423e9f05dcba4d859988" candidate_uuid: "e253914ad79b48bb804cf133415f9fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7c5bc147af492e9eaeea1117c3de94"
I20260806 03:42:16.471756 454 raft_consensus.cc:3037] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:16.471760 599 raft_consensus.cc:3037] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:16.472391 599 raft_consensus.cc:2454] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e253914ad79b48bb804cf133415f9fa6 in term 1.
I20260806 03:42:16.472505 454 raft_consensus.cc:2454] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e253914ad79b48bb804cf133415f9fa6 in term 1.
I20260806 03:42:16.472594 32712 leader_election.cc:302] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab37a34c5fd54779b4864053681605bc, e253914ad79b48bb804cf133415f9fa6; no voters:
I20260806 03:42:16.472708 767 raft_consensus.cc:2781] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:16.472909 767 raft_consensus.cc:686] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [term 1 LEADER]: Becoming Leader. State: Replica: e253914ad79b48bb804cf133415f9fa6, State: Running, Role: LEADER
I20260806 03:42:16.472999 767 consensus_queue.cc:237] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } }
I20260806 03:42:16.473765 32619 catalog_manager.cc:5495] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e253914ad79b48bb804cf133415f9fa6 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "e253914ad79b48bb804cf133415f9fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e253914ad79b48bb804cf133415f9fa6" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:16.492697 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.490178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.30.24.1:44421)
0806 03:42:16.490378 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.490380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.490391 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:16.490432 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.490433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.490451 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.490524 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:16.490623 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.491133 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.491853 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.492065 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.492124 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.492265 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:16.492389 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:16.492529 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.492597 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.492617 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":132,"threads_started":1}
I20260806 03:42:16.497362 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.495274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:40697 (local address 127.30.24.3:40771)
0806 03:42:16.495319 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.495321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.495332 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:16.495387 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.495388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.495397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.495445 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:16.495550 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.495987 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.496775 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.496956 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.497009 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.497162 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:16.497187 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:16.497225 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.497288 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.497308 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:42:16.497385 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.495116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40771 (local address 127.30.24.1:40697)
0806 03:42:16.495283 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:16.495294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:16.495345 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:16.495459 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:16.495460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:16.495461 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:16.495511 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.495512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.496041 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.496042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.496703 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.496705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.497035 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.497036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.497080 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.497094 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:16.497248 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:16.497250 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:16.497284 (+ 34us) client_negotiation.cc:240] Negotiation successful
0806 03:42:16.497319 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260806 03:42:16.499679 454 raft_consensus.cc:1261] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Refusing update from remote peer e253914ad79b48bb804cf133415f9fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:16.499670 599 raft_consensus.cc:1261] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Refusing update from remote peer e253914ad79b48bb804cf133415f9fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:16.499954 784 consensus_queue.cc:1035] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7c5bc147af492e9eaeea1117c3de94" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:16.500087 767 consensus_queue.cc:1035] T cdf7e179c006423e9f05dcba4d859988 P e253914ad79b48bb804cf133415f9fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab37a34c5fd54779b4864053681605bc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:16.507583 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.504933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:33967 (local address 127.30.24.1:44421)
0806 03:42:16.504971 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:16.504973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:16.505114 (+ 141us) server_negotiation.cc:407] Connection header received
0806 03:42:16.505191 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:16.505192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:16.505204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:16.505278 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:16.505422 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.505990 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.506760 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.507047 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.507136 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.507329 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:16.507362 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:16.507432 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:16.507493 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 03:42:16.507514 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 03:42:16.507625 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:16.504901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44421 (local address 127.30.24.3:33967)
0806 03:42:16.505057 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:16.505068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:16.505131 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:16.505298 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:16.505299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:16.505301 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:16.505351 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.505352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.506060 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.506060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.506667 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:16.506669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:16.507171 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:16.507172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:16.507248 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:16.507252 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:16.507433 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:16.507435 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:16.507489 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 03:42:16.507538 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":108,"threads_started":1}
W20260806 03:42:16.507939 735 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 03:42:18.032889 32619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:18.033001 32619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:18.033901 32619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78c61a95768d471f8f68f3eb706a8bc3: Sending DeleteTablet for 3 replicas of tablet cdf7e179c006423e9f05dcba4d859988
I20260806 03:42:18.034164 435 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf7e179c006423e9f05dcba4d859988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:18 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:35214
I20260806 03:42:18.034162 32760 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf7e179c006423e9f05dcba4d859988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:18 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33138
I20260806 03:42:18.034354 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 32694
I20260806 03:42:18.034363 578 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf7e179c006423e9f05dcba4d859988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:18 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:58964
I20260806 03:42:18.035566 822 tablet_replica.cc:330] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: stopping tablet replica
I20260806 03:42:18.035642 822 raft_consensus.cc:2229] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:18.035684 822 raft_consensus.cc:2258] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:18.036301 822 ts_tablet_manager.cc:1905] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:18.037348 820 tablet_replica.cc:330] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: stopping tablet replica
I20260806 03:42:18.037423 820 raft_consensus.cc:2229] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:18.037468 820 raft_consensus.cc:2258] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:18.037729 820 ts_tablet_manager.cc:1905] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:18.038286 822 ts_tablet_manager.cc:1918] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 03:42:18.038350 822 log.cc:1196] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/cdf7e179c006423e9f05dcba4d859988
I20260806 03:42:18.038585 822 ts_tablet_manager.cc:1939] T cdf7e179c006423e9f05dcba4d859988 P ab37a34c5fd54779b4864053681605bc: Deleting consensus metadata
I20260806 03:42:18.038975 32606 catalog_manager.cc:4841] TS ab37a34c5fd54779b4864053681605bc (127.30.24.3:40771): tablet cdf7e179c006423e9f05dcba4d859988 (table test-table [id=aeb206e199f34a9f8575e8328a51bf12]) successfully deleted
I20260806 03:42:18.039446 820 ts_tablet_manager.cc:1918] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 03:42:18.039499 820 log.cc:1196] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/cdf7e179c006423e9f05dcba4d859988
I20260806 03:42:18.039711 820 ts_tablet_manager.cc:1939] T cdf7e179c006423e9f05dcba4d859988 P ce7c5bc147af492e9eaeea1117c3de94: Deleting consensus metadata
I20260806 03:42:18.040150 32607 catalog_manager.cc:4841] TS ce7c5bc147af492e9eaeea1117c3de94 (127.30.24.2:36629): tablet cdf7e179c006423e9f05dcba4d859988 (table test-table [id=aeb206e199f34a9f8575e8328a51bf12]) successfully deleted
W20260806 03:42:18.045504 32605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:44421 (error 108)
I20260806 03:42:18.045806 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 368
I20260806 03:42:18.047103 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:18.045983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44421 (local address 127.0.0.1:45118)
0806 03:42:18.047002 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:18.047058 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:44421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":67,"threads_started":1}
W20260806 03:42:18.047227 32605 catalog_manager.cc:4584] TS e253914ad79b48bb804cf133415f9fa6 (127.30.24.1:44421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdf7e179c006423e9f05dcba4d859988: Network error: Client connection negotiation failed: client connection to 127.30.24.1:44421: connect: Connection refused (error 111)
I20260806 03:42:18.052424 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 512
I20260806 03:42:18.059850 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 32588
2026-08-06T03:42:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5806 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:18 dist-test-slave-10pc krb5kdc[827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:18 dist-test-slave-10pc krb5kdc[827](info): set up 2 sockets
Aug 06 03:42:18 dist-test-slave-10pc krb5kdc[827](info): commencing operation
krb5kdc: starting...
W20260806 03:42:19.129318 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:19.255066 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40357
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:40357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:19.338002 841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:19.338227 841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:19.338253 841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:19.339346 841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:19.339382 841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:19.339396 841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:19.339409 841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:19.339421 841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:19.340890 841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:40357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:19.341104 841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:19.341289 841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:19.346333 841 server_base.cc:1034] running on GCE node
W20260806 03:42:19.346313 847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.346258 849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.346258 846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:19.346796 841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:19.347098 841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:19.348296 841 hybrid_clock.cc:648] HybridClock initialized: now 1785987739348278 us; error 28 us; skew 500 ppm
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:19.350863 841 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:19.351362 841 webserver.cc:460] Webserver started at http://127.30.24.62:40441/ using document root <none> and password file <none>
I20260806 03:42:19.351608 841 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:19.351681 841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:19.351845 841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:19.353003 841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "19373387113548878c4cea52a99afc9f"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "033cee0e57b56087a8ffa634d478d5ce"
server_key_iv: "b855e49c1749a449dd024cb06c9ece29"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.353394 841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "19373387113548878c4cea52a99afc9f"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "033cee0e57b56087a8ffa634d478d5ce"
server_key_iv: "b855e49c1749a449dd024cb06c9ece29"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.354842 841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:19.355615 856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.355798 841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:19.355908 841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "19373387113548878c4cea52a99afc9f"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "033cee0e57b56087a8ffa634d478d5ce"
server_key_iv: "b855e49c1749a449dd024cb06c9ece29"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.355988 841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:19.363986 841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:19.365988 841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:19.366158 841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:19.370414 841 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:40357
I20260806 03:42:19.370473 909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:40357 every 8 connection(s)
I20260806 03:42:19.370829 841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:19.371692 910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:19.374079 910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f: Bootstrap starting.
I20260806 03:42:19.374682 910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:19.374989 910 log.cc:824] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:19.375799 910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f: No bootstrap required, opened a new log
I20260806 03:42:19.377547 910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19373387113548878c4cea52a99afc9f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40357 } }
I20260806 03:42:19.377692 910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:19.377714 910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19373387113548878c4cea52a99afc9f, State: Initialized, Role: FOLLOWER
I20260806 03:42:19.377887 910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19373387113548878c4cea52a99afc9f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40357 } }
I20260806 03:42:19.377959 910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:19.378047 910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:19.378134 910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:19.378698 910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19373387113548878c4cea52a99afc9f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40357 } }
I20260806 03:42:19.378790 910 leader_election.cc:302] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19373387113548878c4cea52a99afc9f; no voters:
I20260806 03:42:19.379009 910 leader_election.cc:288] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:19.379033 913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:19.379415 910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:19.379663 913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [term 1 LEADER]: Becoming Leader. State: Replica: 19373387113548878c4cea52a99afc9f, State: Running, Role: LEADER
I20260806 03:42:19.379823 913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19373387113548878c4cea52a99afc9f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40357 } }
I20260806 03:42:19.380231 915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19373387113548878c4cea52a99afc9f. Latest consensus state: current_term: 1 leader_uuid: "19373387113548878c4cea52a99afc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19373387113548878c4cea52a99afc9f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40357 } } }
I20260806 03:42:19.380313 915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:19.380411 914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19373387113548878c4cea52a99afc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19373387113548878c4cea52a99afc9f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40357 } } }
I20260806 03:42:19.380503 914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:19.380596 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 841
I20260806 03:42:19.380707 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:19.380710 922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:19.380908 30816 external_mini_cluster.cc:1428] Setting key 2916c4247d9f4aad82d58c1efe52ffe4
I20260806 03:42:19.381253 922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:19.383078 922 catalog_manager.cc:1347] Generated new cluster ID: e33642b561814a51950276ebc8ec48bf
I20260806 03:42:19.383138 922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:19.393966 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.381453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.30.24.62:40357)
0806 03:42:19.382078 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:19.382090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:19.384714 (+ 2624us) server_negotiation.cc:407] Connection header received
0806 03:42:19.384882 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:19.384888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:19.384939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:19.385089 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:19.385782 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.386718 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.387510 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.387956 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.388080 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.390520 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:19.390524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:19.390525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:19.390537 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:19.392283 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.392711 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.392714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.392714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.392814 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.393034 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.393035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.393036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.393292 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:19.393492 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:19.393575 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 03:42:19.393732 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":557,"thread_start_us":73,"threads_started":1}
I20260806 03:42:19.410171 922 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:19.410991 922 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:19.419942 922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19373387113548878c4cea52a99afc9f: Generated new TSK 0
I20260806 03:42:19.420142 922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:19.452682 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40357
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:19.539079 936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:19.539297 936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:19.539322 936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:19.540542 936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:19.540584 936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:19.540639 936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:19.542121 936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:19.542423 936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:19.542691 936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:19.548039 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.548066 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.548032 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:19.548309 936 server_base.cc:1034] running on GCE node
I20260806 03:42:19.548614 936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:19.548910 936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:19.550100 936 hybrid_clock.cc:648] HybridClock initialized: now 1785987739550073 us; error 55 us; skew 500 ppm
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:19.552912 936 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:19.553377 936 webserver.cc:460] Webserver started at http://127.30.24.1:39651/ using document root <none> and password file <none>
I20260806 03:42:19.553582 936 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:19.553625 936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:19.553716 936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:19.554780 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "fdd6a415e63a46deb81b3d599a6d4cc0"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "f35ec4bc197b2f029357c13092ad9270"
server_key_iv: "29f6f2b248c727b790710de9f247a5f7"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.555106 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "fdd6a415e63a46deb81b3d599a6d4cc0"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "f35ec4bc197b2f029357c13092ad9270"
server_key_iv: "29f6f2b248c727b790710de9f247a5f7"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.556449 936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:19.557297 952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.557492 936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:19.557569 936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "fdd6a415e63a46deb81b3d599a6d4cc0"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "f35ec4bc197b2f029357c13092ad9270"
server_key_iv: "29f6f2b248c727b790710de9f247a5f7"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.557626 936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:19.587319 936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:19.589776 936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:19.589955 936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:19.590219 936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:19.590574 936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:19.590622 936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.590675 936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:19.590703 936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.596763 936 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:39501
I20260806 03:42:19.596859 1066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:39501 every 8 connection(s)
I20260806 03:42:19.597216 936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:19.598907 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 936
I20260806 03:42:19.599035 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:19.599247 30816 external_mini_cluster.cc:1428] Setting key d974ee9633510528b97deb1ab887b85a
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:19.607996 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.598103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:58139 (local address 127.30.24.62:40357)
0806 03:42:19.598191 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:19.598193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:19.598310 (+ 117us) server_negotiation.cc:407] Connection header received
0806 03:42:19.599034 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:19.599036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:19.599050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:19.599125 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:19.600395 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.600983 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.602021 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.602238 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.602314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.604941 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:19.604942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:19.604943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:19.604953 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:19.606519 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.606961 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.606963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.606963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.607028 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.607305 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.607306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.607307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.607429 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:19.607509 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:19.607802 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 03:42:19.607901 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 03:42:19.608192 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.598016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40357 (local address 127.30.24.1:58139)
0806 03:42:19.598170 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:19.598221 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:19.598880 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:19.599244 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:19.599248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:19.600059 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:19.600290 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:19.600296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.601088 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.601090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:19.601923 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:19.601925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.602336 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.602337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:19.602472 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.603100 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:19.603110 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:19.604839 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:19.606626 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:19.606635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:19.606638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:19.606882 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:19.607103 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:19.607104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:19.607104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:19.607229 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:19.607500 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:19.607509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:19.607658 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 03:42:19.607766 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 03:42:19.607933 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260806 03:42:19.608798 1067 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40357
I20260806 03:42:19.608935 1067 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:19.609179 1067 heartbeater.cc:511] Master 127.30.24.62:40357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
I20260806 03:42:19.609802 873 ts_manager.cc:194] Registered new tserver with Master: fdd6a415e63a46deb81b3d599a6d4cc0 (127.30.24.1:39501)
I20260806 03:42:19.610440 873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:58139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:19.630766 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40357
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:19.716802 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:19.717024 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:19.717051 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:19.718190 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:19.718232 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:19.718266 1072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:19.719677 1072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:19.719884 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:19.720067 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:19.725304 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.725271 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.725271 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:19.725618 1072 server_base.cc:1034] running on GCE node
I20260806 03:42:19.725883 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:19.726148 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:19.727337 1072 hybrid_clock.cc:648] HybridClock initialized: now 1785987739727331 us; error 48 us; skew 500 ppm
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:19.730060 1072 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:19.730523 1072 webserver.cc:460] Webserver started at http://127.30.24.2:38729/ using document root <none> and password file <none>
I20260806 03:42:19.730785 1072 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:19.730856 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:19.731009 1072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:19.732125 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "6e3b4e8e5c7149ffb5ba2998037484f9"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "58fc1da245eb283be6ca4b1175a955cb"
server_key_iv: "4e199d477498f8430a5b29a27afbc4ec"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.732584 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3b4e8e5c7149ffb5ba2998037484f9"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "58fc1da245eb283be6ca4b1175a955cb"
server_key_iv: "4e199d477498f8430a5b29a27afbc4ec"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.733970 1072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:19.734730 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.734912 1072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:19.735016 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "6e3b4e8e5c7149ffb5ba2998037484f9"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "58fc1da245eb283be6ca4b1175a955cb"
server_key_iv: "4e199d477498f8430a5b29a27afbc4ec"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.735106 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:19.751573 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:19.753898 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:19.754071 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:19.754323 1072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:19.754664 1072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:19.754725 1072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.754772 1072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:19.754806 1072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.761123 1072 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:33805
I20260806 03:42:19.761168 1203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:33805 every 8 connection(s)
I20260806 03:42:19.761546 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:19.766746 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1072
I20260806 03:42:19.766868 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:19.767069 30816 external_mini_cluster.cc:1428] Setting key 72d637886fc10211cce0613b5f837fe1
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:19.771914 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.762374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56045 (local address 127.30.24.62:40357)
0806 03:42:19.762450 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:19.762452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:19.762646 (+ 194us) server_negotiation.cc:407] Connection header received
0806 03:42:19.763134 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:19.763135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:19.763146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:19.763206 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:19.764156 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.764737 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.765771 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.765967 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.766026 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.768729 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:19.768731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:19.768731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:19.768740 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:19.770457 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.770878 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.770880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.770880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.770951 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.771215 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.771216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.771217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.771349 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:19.771427 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:19.771724 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 03:42:19.771821 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 03:42:19.772121 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.762342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40357 (local address 127.30.24.2:56045)
0806 03:42:19.762506 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:19.762549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:19.763045 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:19.763259 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:19.763263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:19.763857 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:19.764068 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:19.764073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.764825 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.764826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:19.765671 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:19.765674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.766054 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.766055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:19.766178 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.766794 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:19.766803 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:19.768609 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:19.770548 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:19.770555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:19.770558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:19.770796 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:19.771021 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:19.771022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:19.771022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:19.771136 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:19.771418 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:19.771425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:19.771579 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 03:42:19.771688 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 03:42:19.771866 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260806 03:42:19.772631 1204 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40357
I20260806 03:42:19.772758 1204 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:19.772974 1204 heartbeater.cc:511] Master 127.30.24.62:40357 requested a full tablet report, sending...
I20260806 03:42:19.773428 873 ts_manager.cc:194] Registered new tserver with Master: 6e3b4e8e5c7149ffb5ba2998037484f9 (127.30.24.2:33805)
I20260806 03:42:19.773998 873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:56045
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:19.796059 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40357
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:19.883562 1209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:19.883773 1209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:19.883796 1209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:19.885043 1209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:19.885087 1209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:19.885136 1209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:19.886605 1209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:19.886850 1209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:19.887107 1209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:19.892184 1215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:19.892156 1214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:19.892201 1209 server_base.cc:1034] running on GCE node
W20260806 03:42:19.892171 1217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:19.892592 1209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:19.892861 1209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:19.894048 1209 hybrid_clock.cc:648] HybridClock initialized: now 1785987739894022 us; error 35 us; skew 500 ppm
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:19.896670 1209 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:19.897106 1209 webserver.cc:460] Webserver started at http://127.30.24.3:41995/ using document root <none> and password file <none>
I20260806 03:42:19.897358 1209 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:19.897430 1209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:19.897578 1209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:19.898665 1209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "bea3453f4c1a44be91933b7b939e8d9b"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "c39d3a67c4ca6725c88512d2e67a8669"
server_key_iv: "b081e8aaa615adf1d9f44ead79f1ce3b"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.899051 1209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "bea3453f4c1a44be91933b7b939e8d9b"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "c39d3a67c4ca6725c88512d2e67a8669"
server_key_iv: "b081e8aaa615adf1d9f44ead79f1ce3b"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.900390 1209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:19.901178 1224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.901350 1209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:19.901460 1209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "bea3453f4c1a44be91933b7b939e8d9b"
format_stamp: "Formatted at 2026-08-06 03:42:19 on dist-test-slave-10pc"
server_key: "c39d3a67c4ca6725c88512d2e67a8669"
server_key_iv: "b081e8aaa615adf1d9f44ead79f1ce3b"
server_key_version: "encryptionkey@0"
I20260806 03:42:19.901535 1209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:19.924429 1209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:19.926955 1209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:19.927131 1209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:19.927397 1209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:19.927727 1209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:19.927790 1209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.927834 1209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:19.927877 1209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:19.933876 1209 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:41367
I20260806 03:42:19.933928 1341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:41367 every 8 connection(s)
I20260806 03:42:19.934262 1209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:42:19 dist-test-slave-10pc krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:19.942256 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1209
I20260806 03:42:19.942374 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:19.942584 30816 external_mini_cluster.cc:1428] Setting key e9b7104deee04d0fe2af38f8cc50ac43
I20260806 03:42:19.943980 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.935002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:34363 (local address 127.30.24.62:40357)
0806 03:42:19.935075 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:19.935077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:19.935284 (+ 207us) server_negotiation.cc:407] Connection header received
0806 03:42:19.935794 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:19.935795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:19.935806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:19.935871 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:19.936944 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.937454 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.938532 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.938725 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.938800 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.941101 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:19.941103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:19.941103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:19.941110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:19.942332 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.942858 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.942859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.942860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.942931 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:19.943275 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:19.943276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:19.943277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:19.943413 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:19.943492 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:19.943822 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 03:42:19.943915 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:42:19.944190 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:19.934959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40357 (local address 127.30.24.3:34363)
0806 03:42:19.935150 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:19.935192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:19.935696 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:19.935954 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:19.935958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:19.936622 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:19.936828 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:19.936833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.937543 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.937544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:19.938452 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:19.938455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:19.938828 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:19.938829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:19.938946 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:19.939510 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:19.939518 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:19.941011 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:19.942453 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:19.942460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:19.942463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:19.942756 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:19.943013 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:19.943013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:19.943014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:19.943193 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:19.943486 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:19.943492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:19.943686 (+ 194us) client_negotiation.cc:769] Sending connection context
0806 03:42:19.943773 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 03:42:19.943958 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260806 03:42:19.944808 1343 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40357
I20260806 03:42:19.944890 1343 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:19.945056 1343 heartbeater.cc:511] Master 127.30.24.62:40357 requested a full tablet report, sending...
I20260806 03:42:19.945375 873 ts_manager.cc:194] Registered new tserver with Master: bea3453f4c1a44be91933b7b939e8d9b (127.30.24.3:41367)
I20260806 03:42:19.945755 873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:34363
I20260806 03:42:19.954562 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 03:42:19.967453 930 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:19.966404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.30.24.62:40357)
0806 03:42:19.966472 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:19.966474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:19.966586 (+ 112us) server_negotiation.cc:407] Connection header received
0806 03:42:19.966653 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:19.966654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:19.966664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:19.966724 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:19.967341 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36826: BlockingRecv error: recv got EOF from 127.0.0.1:36826 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 03:42:19.968343 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 936
I20260806 03:42:19.974676 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1072
I20260806 03:42:19.981652 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1209
I20260806 03:42:19.987452 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 841
2026-08-06T03:42:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1926 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:20 dist-test-slave-10pc krb5kdc[1355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:20 dist-test-slave-10pc krb5kdc[1355](info): set up 2 sockets
Aug 06 03:42:20 dist-test-slave-10pc krb5kdc[1355](info): commencing operation
krb5kdc: starting...
W20260806 03:42:21.039168 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:21 dist-test-slave-10pc krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:21.167389 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:32971
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:21.251191 1369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:21.251402 1369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:21.251426 1369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:21.252565 1369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:21.252601 1369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:21.252616 1369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:21.252628 1369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:21.252641 1369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:21.254065 1369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:21.254263 1369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:21.254436 1369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:21.259404 1369 server_base.cc:1034] running on GCE node
W20260806 03:42:21.259349 1377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:21.259361 1375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:21.259382 1374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:21.259778 1369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:21.260105 1369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:21.261308 1369 hybrid_clock.cc:648] HybridClock initialized: now 1785987741261286 us; error 52 us; skew 500 ppm
Aug 06 03:42:21 dist-test-slave-10pc krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987741, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:21.267786 1369 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:21.268340 1369 webserver.cc:460] Webserver started at http://127.30.24.62:39171/ using document root <none> and password file <none>
I20260806 03:42:21.268632 1369 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:21.268709 1369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:21.268851 1369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:21.269991 1369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "145c21d3b3154411bac11ffdeb054535"
format_stamp: "Formatted at 2026-08-06 03:42:21 on dist-test-slave-10pc"
server_key: "fee11a5ccd4b88c9d31229d3a5c571fc"
server_key_iv: "3829321135645a01317d1d150550406f"
server_key_version: "encryptionkey@0"
I20260806 03:42:21.270385 1369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "145c21d3b3154411bac11ffdeb054535"
format_stamp: "Formatted at 2026-08-06 03:42:21 on dist-test-slave-10pc"
server_key: "fee11a5ccd4b88c9d31229d3a5c571fc"
server_key_iv: "3829321135645a01317d1d150550406f"
server_key_version: "encryptionkey@0"
I20260806 03:42:21.271761 1369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:21.272661 1384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:21.272861 1369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:21.272974 1369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "145c21d3b3154411bac11ffdeb054535"
format_stamp: "Formatted at 2026-08-06 03:42:21 on dist-test-slave-10pc"
server_key: "fee11a5ccd4b88c9d31229d3a5c571fc"
server_key_iv: "3829321135645a01317d1d150550406f"
server_key_version: "encryptionkey@0"
I20260806 03:42:21.273062 1369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:21.293020 1369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:21.295531 1369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:21.295692 1369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:21.299921 1369 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:32971
I20260806 03:42:21.299965 1436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:32971 every 8 connection(s)
I20260806 03:42:21.300612 1369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:21.301407 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.303056 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1369
I20260806 03:42:21.303164 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:21.303332 30816 external_mini_cluster.cc:1428] Setting key d4cb3076e761a2e3f93803f98fef5bd6
I20260806 03:42:21.304315 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535: Bootstrap starting.
I20260806 03:42:21.305145 1437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.305536 1437 log.cc:824] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:21.306337 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535: No bootstrap required, opened a new log
I20260806 03:42:21.308737 1437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c21d3b3154411bac11ffdeb054535" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 32971 } }
I20260806 03:42:21.308929 1437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.308981 1437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145c21d3b3154411bac11ffdeb054535, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.309141 1437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c21d3b3154411bac11ffdeb054535" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 32971 } }
I20260806 03:42:21.309242 1437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:21.309288 1437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:21.309330 1437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:21.309894 1437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c21d3b3154411bac11ffdeb054535" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 32971 } }
I20260806 03:42:21.309990 1437 leader_election.cc:302] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 145c21d3b3154411bac11ffdeb054535; no voters:
I20260806 03:42:21.310220 1442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:21.310233 1437 leader_election.cc:288] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:21.310379 1442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [term 1 LEADER]: Becoming Leader. State: Replica: 145c21d3b3154411bac11ffdeb054535, State: Running, Role: LEADER
I20260806 03:42:21.310479 1442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c21d3b3154411bac11ffdeb054535" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 32971 } }
I20260806 03:42:21.310541 1437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:21.310804 1442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "145c21d3b3154411bac11ffdeb054535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c21d3b3154411bac11ffdeb054535" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 32971 } } }
I20260806 03:42:21.310822 1443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 145c21d3b3154411bac11ffdeb054535. Latest consensus state: current_term: 1 leader_uuid: "145c21d3b3154411bac11ffdeb054535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c21d3b3154411bac11ffdeb054535" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 32971 } } }
I20260806 03:42:21.310876 1442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:21.310922 1443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535 [sys.catalog]: This master's current role is: LEADER
Aug 06 03:42:21 dist-test-slave-10pc krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:21.311131 1447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:21.311839 1447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:21.313558 1447 catalog_manager.cc:1347] Generated new cluster ID: 7128db5d4eff4646b401a4c7c4b48b03
I20260806 03:42:21.313889 1447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:21.315316 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:21.303917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.30.24.62:32971)
0806 03:42:21.304096 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:21.304105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:21.304141 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:42:21.304624 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:21.304631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:21.304683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:21.304821 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:21.305749 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:21.307099 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:21.307856 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:21.308574 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:21.308693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:21.311571 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:21.311576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:21.311577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:21.311593 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:21.313641 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:21.314076 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:21.314080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:21.314080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:21.314181 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:21.314400 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:21.314401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:21.314401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:21.314652 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:21.314809 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:21.314944 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 03:42:21.315089 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260806 03:42:21.316941 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:42:21.334815 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:21.330418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.30.24.62:32971)
0806 03:42:21.330465 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:21.330467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:21.330656 (+ 189us) server_negotiation.cc:407] Connection header received
0806 03:42:21.330689 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:21.330690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:21.330701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:21.330765 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:21.331188 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:21.332028 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:21.333082 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:21.333442 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:21.333520 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:21.334283 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:21.334285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:21.334285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:21.334533 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:21.334553 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:21.334561 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:21.334626 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:21.334687 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 03:42:21.334716 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:21.337107 1447 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:21.337651 1447 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:21.340756 1447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 145c21d3b3154411bac11ffdeb054535: Generated new TSK 0
I20260806 03:42:21.340951 1447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:21.344130 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1369
2026-08-06T03:42:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1355 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:21 dist-test-slave-10pc krb5kdc[1472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:21 dist-test-slave-10pc krb5kdc[1472](info): set up 2 sockets
Aug 06 03:42:21 dist-test-slave-10pc krb5kdc[1472](info): commencing operation
krb5kdc: starting...
W20260806 03:42:22.413609 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:22 dist-test-slave-10pc krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:22.547076 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34419
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:34419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:22.631515 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:22.631728 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:22.631758 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:22.632862 1485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:22.632895 1485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:22.632910 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:22.632922 1485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:22.632934 1485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:22.634361 1485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:34419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:22.634557 1485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:22.634729 1485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:22.639605 1485 server_base.cc:1034] running on GCE node
W20260806 03:42:22.639564 1494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:22.639573 1491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:22.639564 1492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:22.640075 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:22.640396 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:22.641593 1485 hybrid_clock.cc:648] HybridClock initialized: now 1785987742641575 us; error 36 us; skew 500 ppm
Aug 06 03:42:22 dist-test-slave-10pc krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:22.644192 1485 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:22.644685 1485 webserver.cc:460] Webserver started at http://127.30.24.62:42873/ using document root <none> and password file <none>
I20260806 03:42:22.644930 1485 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:22.645011 1485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:22.645165 1485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:22.646214 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "bbea29f1a36a4d57b3956e98a7e6bac9"
format_stamp: "Formatted at 2026-08-06 03:42:22 on dist-test-slave-10pc"
server_key: "333bfedc285da4d0f2afbc79b9859c7e"
server_key_iv: "8b3e13c4bcbecc215122abeed2127ea0"
server_key_version: "encryptionkey@0"
I20260806 03:42:22.646608 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "bbea29f1a36a4d57b3956e98a7e6bac9"
format_stamp: "Formatted at 2026-08-06 03:42:22 on dist-test-slave-10pc"
server_key: "333bfedc285da4d0f2afbc79b9859c7e"
server_key_iv: "8b3e13c4bcbecc215122abeed2127ea0"
server_key_version: "encryptionkey@0"
I20260806 03:42:22.647943 1485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:22.648813 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:22.648983 1485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:22.649093 1485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "bbea29f1a36a4d57b3956e98a7e6bac9"
format_stamp: "Formatted at 2026-08-06 03:42:22 on dist-test-slave-10pc"
server_key: "333bfedc285da4d0f2afbc79b9859c7e"
server_key_iv: "8b3e13c4bcbecc215122abeed2127ea0"
server_key_version: "encryptionkey@0"
I20260806 03:42:22.649185 1485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:22.680508 1485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:22.682708 1485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:22.682874 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:22.687180 1485 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:34419
I20260806 03:42:22.687220 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:34419 every 8 connection(s)
I20260806 03:42:22.687618 1485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:22.688886 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.691236 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9: Bootstrap starting.
I20260806 03:42:22.691844 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.692133 1555 log.cc:824] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:22.692840 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1485
I20260806 03:42:22.692921 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9: No bootstrap required, opened a new log
I20260806 03:42:22.692955 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:22.693137 30816 external_mini_cluster.cc:1428] Setting key 1911d4f602778efad885965393afb654
I20260806 03:42:22.694871 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34419 } }
I20260806 03:42:22.695046 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.695086 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbea29f1a36a4d57b3956e98a7e6bac9, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.695305 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34419 } }
I20260806 03:42:22.695410 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:22.695446 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:22.695500 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.696617 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34419 } }
I20260806 03:42:22.696748 1555 leader_election.cc:302] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbea29f1a36a4d57b3956e98a7e6bac9; no voters:
I20260806 03:42:22.696945 1555 leader_election.cc:288] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:22.696970 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.697088 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [term 1 LEADER]: Becoming Leader. State: Replica: bbea29f1a36a4d57b3956e98a7e6bac9, State: Running, Role: LEADER
I20260806 03:42:22.697234 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:22.697237 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34419 } }
I20260806 03:42:22.697667 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34419 } } }
I20260806 03:42:22.697685 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbea29f1a36a4d57b3956e98a7e6bac9. Latest consensus state: current_term: 1 leader_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbea29f1a36a4d57b3956e98a7e6bac9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34419 } } }
I20260806 03:42:22.697921 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:22.698123 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:22.698406 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:22.698930 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:22.700747 1569 catalog_manager.cc:1347] Generated new cluster ID: 709887d2676c4fa4b5d6685e079c852e
I20260806 03:42:22.700801 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:22 dist-test-slave-10pc krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:22.706007 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:22.693713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.30.24.62:34419)
0806 03:42:22.693908 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:22.693915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:22.693941 (+ 26us) server_negotiation.cc:407] Connection header received
0806 03:42:22.694862 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:22.694870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:22.694919 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:22.695035 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:22.695910 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:22.697061 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:22.698238 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:22.698860 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:22.698989 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:22.701940 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:22.701944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:22.701945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:22.701957 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:22.704037 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:22.704503 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:22.704507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:22.704507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:22.704644 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:22.704863 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:22.704864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:22.704864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:22.705219 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:22.705403 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:22.705485 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 03:42:22.705633 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260806 03:42:22.710548 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:22.711086 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:22.720481 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbea29f1a36a4d57b3956e98a7e6bac9: Generated new TSK 0
I20260806 03:42:22.720688 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:22.751253 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34419
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:22.855234 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:22.855469 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:22.855494 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:22.856655 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:22.856693 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:22.856720 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:22.858139 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:22.858353 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:22.858552 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:22.863683 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:22.863719 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:22.863732 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:22.863977 1581 server_base.cc:1034] running on GCE node
I20260806 03:42:22.864200 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:22.864491 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:22.865658 1581 hybrid_clock.cc:648] HybridClock initialized: now 1785987742865642 us; error 27 us; skew 500 ppm
Aug 06 03:42:22 dist-test-slave-10pc krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:22.868394 1581 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:22.868847 1581 webserver.cc:460] Webserver started at http://127.30.24.1:37197/ using document root <none> and password file <none>
I20260806 03:42:22.869036 1581 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:22.869077 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:22.869179 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:22.870216 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "315bf61b18124ea39063758b2a84e6ff"
format_stamp: "Formatted at 2026-08-06 03:42:22 on dist-test-slave-10pc"
server_key: "5d977a961ce39780a2cc06d931946b8e"
server_key_iv: "d19b702977cd0cc5e60741c5f21fa5bd"
server_key_version: "encryptionkey@0"
I20260806 03:42:22.870532 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "315bf61b18124ea39063758b2a84e6ff"
format_stamp: "Formatted at 2026-08-06 03:42:22 on dist-test-slave-10pc"
server_key: "5d977a961ce39780a2cc06d931946b8e"
server_key_iv: "d19b702977cd0cc5e60741c5f21fa5bd"
server_key_version: "encryptionkey@0"
I20260806 03:42:22.871795 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:22.872675 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:22.872867 1581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:22.872972 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "315bf61b18124ea39063758b2a84e6ff"
format_stamp: "Formatted at 2026-08-06 03:42:22 on dist-test-slave-10pc"
server_key: "5d977a961ce39780a2cc06d931946b8e"
server_key_iv: "d19b702977cd0cc5e60741c5f21fa5bd"
server_key_version: "encryptionkey@0"
I20260806 03:42:22.873061 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:22.890285 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:22.892782 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:22.892952 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:22.893229 1581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:22.893563 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:22.893617 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:22.893657 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:22.893689 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:22.899752 1581 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:35849
I20260806 03:42:22.899804 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:35849 every 8 connection(s)
I20260806 03:42:22.900117 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:42:22 dist-test-slave-10pc krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:22.908012 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1581
I20260806 03:42:22.908156 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:22.908368 30816 external_mini_cluster.cc:1428] Setting key 77bd50bc36c9bdaa88e62cf31bbe41a4
I20260806 03:42:22.909272 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:22.900961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:42221 (local address 127.30.24.62:34419)
0806 03:42:22.901022 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:22.901024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:22.901117 (+ 93us) server_negotiation.cc:407] Connection header received
0806 03:42:22.901617 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:22.901618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:22.901630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:22.901693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:22.902662 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:22.903178 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:22.904133 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:22.904346 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:22.904396 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:22.906448 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:22.906450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:22.906450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:22.906457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:22.907679 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:22.908075 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:22.908077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:22.908077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:22.908154 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:22.908446 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:22.908448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:22.908448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:22.908619 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:22.908728 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:22.909061 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 03:42:22.909153 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 03:42:22.909474 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:22.900807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34419 (local address 127.30.24.1:42221)
0806 03:42:22.900986 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:22.901029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:22.901534 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:22.901805 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:22.901809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:22.902381 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:22.902579 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:22.902583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:22.903261 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:22.903261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:22.904059 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:22.904061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:22.904421 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:22.904421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:22.904541 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:22.905011 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:22.905019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:22.906359 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:22.907759 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:22.907766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:22.907768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:22.907982 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:22.908247 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:22.908247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:22.908248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:22.908365 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:22.908735 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:22.908744 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:22.908903 (+ 159us) client_negotiation.cc:769] Sending connection context
0806 03:42:22.909018 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 03:42:22.909245 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260806 03:42:22.910004 1713 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34419
I20260806 03:42:22.910100 1713 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:22.910295 1713 heartbeater.cc:511] Master 127.30.24.62:34419 requested a full tablet report, sending...
I20260806 03:42:22.910843 1518 ts_manager.cc:194] Registered new tserver with Master: 315bf61b18124ea39063758b2a84e6ff (127.30.24.1:35849)
I20260806 03:42:22.911459 1518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:42221
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:22.935837 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34419
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:23.019985 1718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:23.020211 1718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:23.020236 1718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:23.021521 1718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:23.021570 1718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:23.021597 1718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:23.023136 1718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:23.023386 1718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:23.023656 1718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:23.028801 1724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:23.028841 1723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:23.028978 1718 server_base.cc:1034] running on GCE node
W20260806 03:42:23.028841 1726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:23.029292 1718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:23.029552 1718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:23.030718 1718 hybrid_clock.cc:648] HybridClock initialized: now 1785987743030708 us; error 55 us; skew 500 ppm
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987743, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:23.033362 1718 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:23.033802 1718 webserver.cc:460] Webserver started at http://127.30.24.2:35749/ using document root <none> and password file <none>
I20260806 03:42:23.034051 1718 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:23.034121 1718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:23.034276 1718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:23.035349 1718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "d87638bfc7d14a7fb7b2a5e7f6eeea1c"
format_stamp: "Formatted at 2026-08-06 03:42:23 on dist-test-slave-10pc"
server_key: "44bca47728a37cdd2b81bb665449c80d"
server_key_iv: "df171a7d275ce298251c17633195fe87"
server_key_version: "encryptionkey@0"
I20260806 03:42:23.035732 1718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "d87638bfc7d14a7fb7b2a5e7f6eeea1c"
format_stamp: "Formatted at 2026-08-06 03:42:23 on dist-test-slave-10pc"
server_key: "44bca47728a37cdd2b81bb665449c80d"
server_key_iv: "df171a7d275ce298251c17633195fe87"
server_key_version: "encryptionkey@0"
I20260806 03:42:23.037248 1718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:23.038028 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:23.038282 1718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:23.038398 1718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "d87638bfc7d14a7fb7b2a5e7f6eeea1c"
format_stamp: "Formatted at 2026-08-06 03:42:23 on dist-test-slave-10pc"
server_key: "44bca47728a37cdd2b81bb665449c80d"
server_key_iv: "df171a7d275ce298251c17633195fe87"
server_key_version: "encryptionkey@0"
I20260806 03:42:23.038477 1718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:23.048291 1718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:23.050820 1718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:23.050983 1718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:23.051303 1718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:23.051638 1718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:23.051695 1718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:23.051739 1718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:23.051771 1718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:23.057792 1718 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:38849
I20260806 03:42:23.057883 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:38849 every 8 connection(s)
I20260806 03:42:23.058221 1718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:23.060925 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1718
I20260806 03:42:23.061054 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:23.061232 30816 external_mini_cluster.cc:1428] Setting key 6e968e5d028956f701ab914c7e63e227
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987743, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:23.068884 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:23.059006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:35303 (local address 127.30.24.62:34419)
0806 03:42:23.059110 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:23.059113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:23.059303 (+ 190us) server_negotiation.cc:407] Connection header received
0806 03:42:23.059866 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:23.059867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:23.059881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:23.059958 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:23.061037 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.061697 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.062773 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.063077 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.063174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:23.065705 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:23.065706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:23.065706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:23.065713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:23.067254 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:23.067802 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:23.067803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:23.067804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:23.067867 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:23.068167 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:23.068168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:23.068168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:23.068289 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:23.068341 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:23.068741 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 03:42:23.068817 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:42:23.069164 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:23.058962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34419 (local address 127.30.24.2:35303)
0806 03:42:23.059174 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:23.059218 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:23.059750 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:23.060031 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:23.060036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:23.060708 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:23.060912 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:23.060917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.061789 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.061790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:23.062678 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:23.062682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.063192 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.063193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:23.063345 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:23.063938 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:23.063947 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:23.065613 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:23.067345 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:23.067353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:23.067356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:23.067714 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:23.067936 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:23.067937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:23.067938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:23.068094 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:23.068361 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:23.068368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:23.068586 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 03:42:23.068719 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 03:42:23.068908 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260806 03:42:23.069744 1851 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34419
I20260806 03:42:23.069857 1851 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:23.070066 1851 heartbeater.cc:511] Master 127.30.24.62:34419 requested a full tablet report, sending...
I20260806 03:42:23.070452 1518 ts_manager.cc:194] Registered new tserver with Master: d87638bfc7d14a7fb7b2a5e7f6eeea1c (127.30.24.2:38849)
I20260806 03:42:23.070843 1518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:35303
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:23.089581 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34419
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:23.173032 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:23.173246 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:23.173269 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:23.174479 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:23.174522 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:23.174549 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:23.175992 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:23.176309 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:23.176581 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:23.181685 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:23.181675 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:23.181675 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:23.181922 1856 server_base.cc:1034] running on GCE node
I20260806 03:42:23.182163 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:23.182411 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:23.183557 1856 hybrid_clock.cc:648] HybridClock initialized: now 1785987743183545 us; error 27 us; skew 500 ppm
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987743, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:23.186260 1856 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:23.186753 1856 webserver.cc:460] Webserver started at http://127.30.24.3:45545/ using document root <none> and password file <none>
I20260806 03:42:23.186964 1856 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:23.187006 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:23.187098 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:23.188133 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "64ad9efde5774ed59b0a1906f602cf9e"
format_stamp: "Formatted at 2026-08-06 03:42:23 on dist-test-slave-10pc"
server_key: "8610a4516518d3985083d7ab33811544"
server_key_iv: "d42a208cb7bf75b9997bef8dfe7f428c"
server_key_version: "encryptionkey@0"
I20260806 03:42:23.188444 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "64ad9efde5774ed59b0a1906f602cf9e"
format_stamp: "Formatted at 2026-08-06 03:42:23 on dist-test-slave-10pc"
server_key: "8610a4516518d3985083d7ab33811544"
server_key_iv: "d42a208cb7bf75b9997bef8dfe7f428c"
server_key_version: "encryptionkey@0"
I20260806 03:42:23.189803 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:23.190564 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:23.190765 1856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:23.190881 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "64ad9efde5774ed59b0a1906f602cf9e"
format_stamp: "Formatted at 2026-08-06 03:42:23 on dist-test-slave-10pc"
server_key: "8610a4516518d3985083d7ab33811544"
server_key_iv: "d42a208cb7bf75b9997bef8dfe7f428c"
server_key_version: "encryptionkey@0"
I20260806 03:42:23.190961 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:23.202939 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:23.204921 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:23.205116 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:23.205360 1856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:23.205711 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:23.205768 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:23.205811 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:23.205838 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:23.212011 1856 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:45027
I20260806 03:42:23.212061 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:45027 every 8 connection(s)
I20260806 03:42:23.212391 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:23.214857 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 1856
I20260806 03:42:23.214941 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:23.215133 30816 external_mini_cluster.cc:1428] Setting key ac3a8e7b4f32f9b27aa9fd8119ab3f6e
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987743, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:23.222113 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:23.213314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:58077 (local address 127.30.24.62:34419)
0806 03:42:23.213376 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:23.213378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:23.213610 (+ 232us) server_negotiation.cc:407] Connection header received
0806 03:42:23.214179 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:23.214180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:23.214194 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:23.214267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:23.215387 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.215905 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.217041 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.217235 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.217320 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:23.219399 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:23.219401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:23.219401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:23.219409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:23.220706 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:23.221121 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:23.221125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:23.221125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:23.221183 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:23.221446 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:23.221447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:23.221448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:23.221565 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:23.221645 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:23.221940 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 03:42:23.222023 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 03:42:23.222303 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:23.213300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34419 (local address 127.30.24.3:58077)
0806 03:42:23.213481 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:23.213520 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:23.214064 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:23.214350 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:23.214354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:23.215032 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:23.215287 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:23.215293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.216009 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.216010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:23.216936 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:23.216938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.217327 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.217328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:23.217467 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:23.217949 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:23.217958 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:23.219297 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:23.220797 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:23.220805 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:23.220807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:23.221031 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:23.221257 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:23.221258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:23.221259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:23.221369 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:23.221634 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:23.221641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:23.221795 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 03:42:23.221903 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 03:42:23.222067 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260806 03:42:23.222803 1990 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34419
I20260806 03:42:23.222909 1990 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:23.223094 1990 heartbeater.cc:511] Master 127.30.24.62:34419 requested a full tablet report, sending...
I20260806 03:42:23.223472 1518 ts_manager.cc:194] Registered new tserver with Master: 64ad9efde5774ed59b0a1906f602cf9e (127.30.24.3:45027)
I20260806 03:42:23.223811 1518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:58077
I20260806 03:42:23.226826 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987743, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987743, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:23.250923 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:23.244001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.30.24.62:34419)
0806 03:42:23.244075 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:23.244078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:23.244189 (+ 111us) server_negotiation.cc:407] Connection header received
0806 03:42:23.244244 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:23.244245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:23.244256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:23.244309 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:23.244963 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.245515 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.246211 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:23.246392 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:23.246442 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:23.248678 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:23.248679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:23.248680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:23.248687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:23.249917 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:23.250227 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:23.250228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:23.250228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:23.250282 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:23.250447 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:23.250447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:23.250448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:23.250593 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:23.250680 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:23.250757 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 03:42:23.250819 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 03:42:23.251233 1518 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42510
I20260806 03:42:23.251703 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1581
I20260806 03:42:23.257614 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1718
I20260806 03:42:23.262532 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1856
I20260806 03:42:23.267702 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 1485
2026-08-06T03:42:23Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1924 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[2003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[2003](info): set up 2 sockets
Aug 06 03:42:23 dist-test-slave-10pc krb5kdc[2003](info): commencing operation
krb5kdc: starting...
W20260806 03:42:24.322515 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:24.452319 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:36799
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:36799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:24.536257 2016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:24.536499 2016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:24.536521 2016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:24.537729 2016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:24.537781 2016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:24.537796 2016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:24.537820 2016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:24.537832 2016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:24.539415 2016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:36799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:36799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:24.539686 2016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:24.539924 2016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:24.544870 2022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:24.544909 2016 server_base.cc:1034] running on GCE node
W20260806 03:42:24.544890 2024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:24.544878 2021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:24.545378 2016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:24.545753 2016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:24.546906 2016 hybrid_clock.cc:648] HybridClock initialized: now 1785987744546889 us; error 36 us; skew 500 ppm
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:24.549518 2016 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:24.550096 2016 webserver.cc:460] Webserver started at http://127.30.24.62:46015/ using document root <none> and password file <none>
I20260806 03:42:24.550360 2016 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:24.550431 2016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:24.550602 2016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:24.551707 2016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "bed32ce990954fb8a01d79ea17f18b0c"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "4c74ac8abd2b7317cc3148f4ee62db2c"
server_key_iv: "f5104dd859e2a9fdcafa20d21c92ad33"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.552086 2016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "bed32ce990954fb8a01d79ea17f18b0c"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "4c74ac8abd2b7317cc3148f4ee62db2c"
server_key_iv: "f5104dd859e2a9fdcafa20d21c92ad33"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.553572 2016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:24.554411 2031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.554607 2016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:24.554718 2016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "bed32ce990954fb8a01d79ea17f18b0c"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "4c74ac8abd2b7317cc3148f4ee62db2c"
server_key_iv: "f5104dd859e2a9fdcafa20d21c92ad33"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.554796 2016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:24.573578 2016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:24.575660 2016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:24.575847 2016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:24.580235 2016 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:36799
I20260806 03:42:24.580289 2086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:36799 every 8 connection(s)
I20260806 03:42:24.580852 2016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:24.581574 2087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:24.583855 2087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c: Bootstrap starting.
I20260806 03:42:24.584399 2087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.584677 2087 log.cc:824] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:24.585357 2087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c: No bootstrap required, opened a new log
I20260806 03:42:24.586882 2087 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed32ce990954fb8a01d79ea17f18b0c" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36799 } }
I20260806 03:42:24.587034 2087 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.587057 2087 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed32ce990954fb8a01d79ea17f18b0c, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.587162 2087 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed32ce990954fb8a01d79ea17f18b0c" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36799 } }
I20260806 03:42:24.587213 2087 raft_consensus.cc:388] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:24.587235 2087 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:24.587270 2087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.588148 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2016
I20260806 03:42:24.588269 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:24.588306 2087 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed32ce990954fb8a01d79ea17f18b0c" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36799 } }
I20260806 03:42:24.588486 30816 external_mini_cluster.cc:1428] Setting key 665e86a09701593de61b62dec448f106
I20260806 03:42:24.588510 2087 leader_election.cc:302] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bed32ce990954fb8a01d79ea17f18b0c; no voters:
I20260806 03:42:24.588708 2087 leader_election.cc:288] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:24.588788 2090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.588972 2087 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:24.588972 2090 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [term 1 LEADER]: Becoming Leader. State: Replica: bed32ce990954fb8a01d79ea17f18b0c, State: Running, Role: LEADER
I20260806 03:42:24.589159 2090 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed32ce990954fb8a01d79ea17f18b0c" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36799 } }
I20260806 03:42:24.589522 2091 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bed32ce990954fb8a01d79ea17f18b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed32ce990954fb8a01d79ea17f18b0c" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36799 } } }
I20260806 03:42:24.589756 2091 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:24.589677 2092 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed32ce990954fb8a01d79ea17f18b0c. Latest consensus state: current_term: 1 leader_uuid: "bed32ce990954fb8a01d79ea17f18b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed32ce990954fb8a01d79ea17f18b0c" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36799 } } }
I20260806 03:42:24.590000 2092 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:24.590078 2096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:24.590734 2096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:24.592315 2096 catalog_manager.cc:1347] Generated new cluster ID: 4b748c43956b461ea44727392a0864f3
I20260806 03:42:24.592378 2096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:24.598742 2096 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:24.599359 2096 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:24.600898 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:24.589156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.30.24.62:36799)
0806 03:42:24.589372 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:24.589385 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:24.589413 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:42:24.589564 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:24.589573 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:24.589631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:24.589787 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:24.590726 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.592158 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.593324 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.593939 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.594038 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:24.596882 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:24.596886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:24.596887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:24.596899 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:24.599070 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:24.599502 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:24.599505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:24.599505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:24.599619 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:24.599858 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:24.599859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:24.599859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:24.600133 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:24.600284 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:24.600405 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 03:42:24.600688 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260806 03:42:24.610198 2096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed32ce990954fb8a01d79ea17f18b0c: Generated new TSK 0
I20260806 03:42:24.610412 2096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:24.639535 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36799
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:24.724867 2113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:24.725085 2113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:24.725109 2113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:24.726217 2113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:24.726262 2113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:24.726289 2113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:24.727752 2113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:24.727944 2113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:24.728138 2113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:24.733208 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:24.733184 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:24.733234 2113 server_base.cc:1034] running on GCE node
W20260806 03:42:24.733180 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:24.733614 2113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:24.733881 2113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:24.735047 2113 hybrid_clock.cc:648] HybridClock initialized: now 1785987744735037 us; error 45 us; skew 500 ppm
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:24.737632 2113 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:24.738108 2113 webserver.cc:460] Webserver started at http://127.30.24.1:36481/ using document root <none> and password file <none>
I20260806 03:42:24.738369 2113 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:24.738438 2113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:24.738590 2113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:24.739674 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "9fb3d8221e044356bb3458424c77cc89"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "933f8589f7e0e637b7e7cbec54689d59"
server_key_iv: "ec065ac2c353713547c54b8db5024c2b"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.740056 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "9fb3d8221e044356bb3458424c77cc89"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "933f8589f7e0e637b7e7cbec54689d59"
server_key_iv: "ec065ac2c353713547c54b8db5024c2b"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.741539 2113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:24.742364 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.742611 2113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:24.742729 2113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "9fb3d8221e044356bb3458424c77cc89"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "933f8589f7e0e637b7e7cbec54689d59"
server_key_iv: "ec065ac2c353713547c54b8db5024c2b"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.742812 2113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:24.761653 2113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:24.763705 2113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:24.763885 2113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:24.764151 2113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:24.764518 2113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:24.764570 2113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.764605 2113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:24.764632 2113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.770557 2113 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:42163
I20260806 03:42:24.770644 2242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:42163 every 8 connection(s)
I20260806 03:42:24.770936 2113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:24.775159 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2113
I20260806 03:42:24.775298 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:24.775502 30816 external_mini_cluster.cc:1428] Setting key b915afa3ddcacc1d9dcde1c67e42b773
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:24.782016 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:24.771637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36799 (local address 127.30.24.1:35323)
0806 03:42:24.771834 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:24.771877 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:24.772978 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:24.773303 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:24.773306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:24.773584 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:24.773780 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:24.773784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.774453 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.774454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:24.775415 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:24.775418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.775953 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.775955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:24.776107 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:24.776703 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:24.776711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:24.778429 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:24.780528 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:24.780536 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:24.780539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:24.780788 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:24.781023 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:24.781024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:24.781024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:24.781165 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:24.781498 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:24.781504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:24.781635 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 03:42:24.781715 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:42:24.781848 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":119,"threads_started":1}
I20260806 03:42:24.782032 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:24.771691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:35323 (local address 127.30.24.62:36799)
0806 03:42:24.771758 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:24.771760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:24.771958 (+ 198us) server_negotiation.cc:407] Connection header received
0806 03:42:24.773147 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:24.773148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:24.773161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:24.773224 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:24.773864 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.774372 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.775519 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.775829 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.775914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:24.778535 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:24.778537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:24.778537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:24.778544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:24.780300 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:24.780873 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:24.780875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:24.780875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:24.780948 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:24.781266 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:24.781268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:24.781268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:24.781430 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:24.781505 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:24.781788 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 03:42:24.781890 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:42:24.782756 2243 heartbeater.cc:347] Connected to a master server at 127.30.24.62:36799
I20260806 03:42:24.782852 2243 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:24.783056 2243 heartbeater.cc:511] Master 127.30.24.62:36799 requested a full tablet report, sending...
I20260806 03:42:24.783675 2050 ts_manager.cc:194] Registered new tserver with Master: 9fb3d8221e044356bb3458424c77cc89 (127.30.24.1:42163)
I20260806 03:42:24.784691 2050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:35323
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:24.803879 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36799
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:24.892333 2248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:24.892596 2248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:24.892621 2248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:24.893862 2248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:24.893910 2248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:24.893939 2248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:24.895514 2248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:24.895777 2248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:24.896027 2248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:24.901443 2254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:24.901401 2256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:24.901429 2253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:24.901594 2248 server_base.cc:1034] running on GCE node
I20260806 03:42:24.901898 2248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:24.902159 2248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:24.903304 2248 hybrid_clock.cc:648] HybridClock initialized: now 1785987744903290 us; error 32 us; skew 500 ppm
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:24.905906 2248 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:24.906373 2248 webserver.cc:460] Webserver started at http://127.30.24.2:44127/ using document root <none> and password file <none>
I20260806 03:42:24.906606 2248 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:24.906657 2248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:24.906838 2248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:24.907941 2248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "066c797162d041f1bf689a7c6d13a132"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "69f6a3abc3364a4f31b71c2969de5b80"
server_key_iv: "7ffbf9814a91aed7ee0d40a506ae15b3"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.908325 2248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "066c797162d041f1bf689a7c6d13a132"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "69f6a3abc3364a4f31b71c2969de5b80"
server_key_iv: "7ffbf9814a91aed7ee0d40a506ae15b3"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.909739 2248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:24.910547 2264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.910764 2248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:24.910859 2248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "066c797162d041f1bf689a7c6d13a132"
format_stamp: "Formatted at 2026-08-06 03:42:24 on dist-test-slave-10pc"
server_key: "69f6a3abc3364a4f31b71c2969de5b80"
server_key_iv: "7ffbf9814a91aed7ee0d40a506ae15b3"
server_key_version: "encryptionkey@0"
I20260806 03:42:24.910938 2248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:24.933856 2248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:24.935825 2248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:24.935990 2248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:24.936246 2248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:24.936621 2248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:24.936666 2248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.936687 2248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:24.936731 2248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:24.942859 2248 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:45501
I20260806 03:42:24.942904 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:45501 every 8 connection(s)
I20260806 03:42:24.943240 2248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:24 dist-test-slave-10pc krb5kdc[2003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:24.950017 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2248
I20260806 03:42:24.950150 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:24.950363 30816 external_mini_cluster.cc:1428] Setting key 43dc8981e91c60651b9d360343f471aa
I20260806 03:42:24.953156 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:24.944027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:50145 (local address 127.30.24.62:36799)
0806 03:42:24.944076 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:24.944078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:24.944305 (+ 227us) server_negotiation.cc:407] Connection header received
0806 03:42:24.945345 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:24.945346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:24.945356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:24.945418 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:24.946096 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.946595 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.947591 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.947778 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.947832 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:24.949949 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:24.949952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:24.949952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:24.949962 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:24.951610 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:24.952068 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:24.952069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:24.952070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:24.952137 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:24.952403 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:24.952404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:24.952404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:24.952596 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:24.952677 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:24.952970 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 03:42:24.953062 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:42:24.953372 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:24.944000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36799 (local address 127.30.24.2:50145)
0806 03:42:24.944173 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:24.944217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:24.945245 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:24.945475 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:24.945478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:24.945809 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:24.946015 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:24.946019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.946710 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.946711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:24.947515 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:24.947518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:24.947855 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:24.947856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:24.947957 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:24.948422 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:24.948431 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:24.949845 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:24.951723 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:24.951731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:24.951734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:24.951983 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:24.952209 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:24.952210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:24.952210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:24.952329 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:24.952669 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:24.952677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:24.952829 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 03:42:24.952946 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 03:42:24.953117 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260806 03:42:24.953905 2380 heartbeater.cc:347] Connected to a master server at 127.30.24.62:36799
I20260806 03:42:24.954020 2380 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:24.954207 2380 heartbeater.cc:511] Master 127.30.24.62:36799 requested a full tablet report, sending...
I20260806 03:42:24.954664 2050 ts_manager.cc:194] Registered new tserver with Master: 066c797162d041f1bf689a7c6d13a132 (127.30.24.2:45501)
I20260806 03:42:24.955190 2050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:50145
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:24.979466 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36799
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:25.066607 2385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:25.066812 2385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:25.066835 2385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:25.067912 2385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:25.067945 2385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:25.067970 2385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:25.069509 2385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:25.069707 2385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:25.069893 2385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:25.075125 2391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:25.075153 2390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:25.075161 2393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:25.075196 2385 server_base.cc:1034] running on GCE node
I20260806 03:42:25.075554 2385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:25.075817 2385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:25.076993 2385 hybrid_clock.cc:648] HybridClock initialized: now 1785987745076979 us; error 47 us; skew 500 ppm
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987745, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:25.079555 2385 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:25.080086 2385 webserver.cc:460] Webserver started at http://127.30.24.3:41883/ using document root <none> and password file <none>
I20260806 03:42:25.080358 2385 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:25.080430 2385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:25.080597 2385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:25.081697 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "c9e8cfb5b0784a6d8430d54ba13b2c91"
format_stamp: "Formatted at 2026-08-06 03:42:25 on dist-test-slave-10pc"
server_key: "a4d973e8e37ef42018038574befd1a8f"
server_key_iv: "27f54c5aa545c2304852d1ce19f20adc"
server_key_version: "encryptionkey@0"
I20260806 03:42:25.082078 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "c9e8cfb5b0784a6d8430d54ba13b2c91"
format_stamp: "Formatted at 2026-08-06 03:42:25 on dist-test-slave-10pc"
server_key: "a4d973e8e37ef42018038574befd1a8f"
server_key_iv: "27f54c5aa545c2304852d1ce19f20adc"
server_key_version: "encryptionkey@0"
I20260806 03:42:25.083479 2385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:25.084251 2400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:25.084434 2385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:25.084582 2385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "c9e8cfb5b0784a6d8430d54ba13b2c91"
format_stamp: "Formatted at 2026-08-06 03:42:25 on dist-test-slave-10pc"
server_key: "a4d973e8e37ef42018038574befd1a8f"
server_key_iv: "27f54c5aa545c2304852d1ce19f20adc"
server_key_version: "encryptionkey@0"
I20260806 03:42:25.084676 2385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:25.094960 2385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:25.097117 2385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:25.097294 2385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:25.097537 2385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:25.097874 2385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:25.097931 2385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:25.097975 2385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:25.098001 2385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:25.103995 2385 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:39099
I20260806 03:42:25.104067 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:39099 every 8 connection(s)
I20260806 03:42:25.104410 2385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:25.105604 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2385
I20260806 03:42:25.105703 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:25.105892 30816 external_mini_cluster.cc:1428] Setting key 8ef359c2c954de0a3229af5e94d730a5
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987745, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:25.114440 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:25.105218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:35757 (local address 127.30.24.62:36799)
0806 03:42:25.105266 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:25.105268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:25.105505 (+ 237us) server_negotiation.cc:407] Connection header received
0806 03:42:25.106874 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:25.106874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:25.106885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:25.106919 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:25.107559 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.108054 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.109120 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.109323 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.109378 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:25.111673 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:25.111674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:25.111675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:25.111681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:25.113185 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:25.113535 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:25.113536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:25.113536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:25.113613 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:25.113892 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:25.113892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:25.113893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:25.114005 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:25.114059 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:25.114305 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 03:42:25.114374 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:25.114590 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:25.105194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36799 (local address 127.30.24.3:35757)
0806 03:42:25.105342 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:25.105377 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:25.106748 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:25.106985 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:25.106988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:25.107282 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:25.107484 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:25.107488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.108135 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.108136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:25.109020 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:25.109022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.109397 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.109397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:25.109504 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:25.110087 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:25.110096 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:25.111586 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:25.113256 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:25.113262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:25.113264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:25.113469 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:25.113696 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:25.113697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:25.113697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:25.113823 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:25.114064 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:25.114069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:25.114191 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 03:42:25.114273 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 03:42:25.114393 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260806 03:42:25.115115 2519 heartbeater.cc:347] Connected to a master server at 127.30.24.62:36799
I20260806 03:42:25.115245 2519 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:25.115432 2519 heartbeater.cc:511] Master 127.30.24.62:36799 requested a full tablet report, sending...
I20260806 03:42:25.115823 2050 ts_manager.cc:194] Registered new tserver with Master: c9e8cfb5b0784a6d8430d54ba13b2c91 (127.30.24.3:39099)
I20260806 03:42:25.116209 2050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:35757
I20260806 03:42:25.117672 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:25.142781 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:25.134541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.30.24.1:42163)
0806 03:42:25.134708 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:25.134714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:25.134725 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:25.134752 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:25.134754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:25.134768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:25.134851 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:25.135531 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.136245 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.137099 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.137432 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.137527 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:25.139873 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:25.139882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:25.139882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:25.139891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:25.141483 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:25.141864 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:25.141868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:25.141868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:25.141945 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:25.142171 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:25.142172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:25.142173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:25.142324 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:25.142497 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:25.142545 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 03:42:25.142641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:25.151094 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:25.144164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.30.24.62:36799)
0806 03:42:25.144231 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:25.144233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:25.144246 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:25.144304 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:25.144305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:25.144314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:25.144360 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:25.144836 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.145411 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.146043 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:25.146260 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:25.146312 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:25.148440 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:25.148441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:25.148442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:25.148470 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:25.149874 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:25.150246 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:25.150248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:25.150248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:25.150310 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:25.150520 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:25.150521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:25.150522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:25.150675 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:25.150792 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:25.150906 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 03:42:25.150982 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 03:42:25.151365 2050 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58962
I20260806 03:42:25.151755 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2113
I20260806 03:42:25.158169 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2248
I20260806 03:42:25.164168 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2385
I20260806 03:42:25.171311 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2016
2026-08-06T03:42:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1903 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2531](info): set up 2 sockets
Aug 06 03:42:25 dist-test-slave-10pc krb5kdc[2531](info): commencing operation
krb5kdc: starting...
W20260806 03:42:26.221803 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:26.355427 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:36171
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:36171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:26.440557 2544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:26.440814 2544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:26.440856 2544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:26.442066 2544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:26.442144 2544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:26.442171 2544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:26.442203 2544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:26.442233 2544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:26.443794 2544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:36171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:36171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:26.444079 2544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:26.444320 2544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:26.449258 2549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.449337 2551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.449258 2553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:26.449548 2544 server_base.cc:1034] running on GCE node
I20260806 03:42:26.449854 2544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:26.450160 2544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:26.451341 2544 hybrid_clock.cc:648] HybridClock initialized: now 1785987746451320 us; error 53 us; skew 500 ppm
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:26.457146 2544 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:26.457504 2544 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:26.457605 2544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:26.457760 2544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:26.458884 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "a94d2d98e1734fa09c6aff742fbc0192"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "4bd48c439bc1e5b5acbaec9c04622272"
server_key_iv: "aa129b37505b4a244d7eb934530c8e4d"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.459268 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "a94d2d98e1734fa09c6aff742fbc0192"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "4bd48c439bc1e5b5acbaec9c04622272"
server_key_iv: "aa129b37505b4a244d7eb934530c8e4d"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.460673 2544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:42:26.461548 2559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.461730 2544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:26.461849 2544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "a94d2d98e1734fa09c6aff742fbc0192"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "4bd48c439bc1e5b5acbaec9c04622272"
server_key_iv: "aa129b37505b4a244d7eb934530c8e4d"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.461934 2544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:26.484381 2544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:26.487093 2544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:26.487282 2544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:26.492125 2544 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:36171
I20260806 03:42:26.492168 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:36171 every 8 connection(s)
I20260806 03:42:26.492604 2544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:26.493770 2615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:26.496172 2615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192: Bootstrap starting.
I20260806 03:42:26.496822 2615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.497143 2615 log.cc:824] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:26.497954 2615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192: No bootstrap required, opened a new log
I20260806 03:42:26.499779 2615 raft_consensus.cc:348] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94d2d98e1734fa09c6aff742fbc0192" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36171 } }
I20260806 03:42:26.499943 2615 raft_consensus.cc:374] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.499969 2615 raft_consensus.cc:729] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94d2d98e1734fa09c6aff742fbc0192, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.500178 2615 consensus_queue.cc:260] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94d2d98e1734fa09c6aff742fbc0192" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36171 } }
I20260806 03:42:26.500247 2615 raft_consensus.cc:388] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:26.500272 2615 raft_consensus.cc:482] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:26.500344 2615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.501302 2615 raft_consensus.cc:504] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94d2d98e1734fa09c6aff742fbc0192" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36171 } }
I20260806 03:42:26.501456 2615 leader_election.cc:302] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a94d2d98e1734fa09c6aff742fbc0192; no voters:
I20260806 03:42:26.501576 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2544
I20260806 03:42:26.501665 2615 leader_election.cc:288] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:26.501705 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:26.501760 2618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.501927 30816 external_mini_cluster.cc:1428] Setting key 61fea669b1ebcf9f8690c6b62e480858
I20260806 03:42:26.501945 2618 raft_consensus.cc:686] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [term 1 LEADER]: Becoming Leader. State: Replica: a94d2d98e1734fa09c6aff742fbc0192, State: Running, Role: LEADER
I20260806 03:42:26.502049 2615 sys_catalog.cc:564] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:26.502038 2618 consensus_queue.cc:237] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94d2d98e1734fa09c6aff742fbc0192" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36171 } }
I20260806 03:42:26.502395 2619 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a94d2d98e1734fa09c6aff742fbc0192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94d2d98e1734fa09c6aff742fbc0192" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36171 } } }
I20260806 03:42:26.502496 2619 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:26.502413 2620 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a94d2d98e1734fa09c6aff742fbc0192. Latest consensus state: current_term: 1 leader_uuid: "a94d2d98e1734fa09c6aff742fbc0192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94d2d98e1734fa09c6aff742fbc0192" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36171 } } }
I20260806 03:42:26.503196 2620 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:26.503322 2624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:26.503939 2624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:26.505786 2624 catalog_manager.cc:1347] Generated new cluster ID: e75f95f9bb484278809035fd1c23a8c8
I20260806 03:42:26.505848 2624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:26.513870 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:26.502719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.30.24.62:36171)
0806 03:42:26.503087 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:26.503097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:26.503133 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:42:26.503313 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:26.503321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:26.503381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:26.503577 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:26.504502 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.505970 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.506765 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.507199 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.507321 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:26.509783 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:26.509787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:26.509787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:26.509799 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:26.512158 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:26.512633 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:26.512636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:26.512636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:26.512733 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:26.512948 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:26.512949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:26.512950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:26.513208 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:26.513377 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:26.513489 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 03:42:26.513645 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":74,"threads_started":1}
I20260806 03:42:26.541391 2624 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:26.542227 2624 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:26.546562 2624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192: Generated new TSK 0
I20260806 03:42:26.546777 2624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:26.583223 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36171
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:26.667179 2641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:26.667382 2641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:26.667405 2641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:26.668581 2641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:26.668629 2641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:26.668656 2641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:26.670159 2641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:26.670418 2641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:26.670660 2641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:26.675796 2649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.675853 2646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:26.675936 2641 server_base.cc:1034] running on GCE node
W20260806 03:42:26.675832 2647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:26.676329 2641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:26.676641 2641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:26.677862 2641 hybrid_clock.cc:648] HybridClock initialized: now 1785987746677834 us; error 53 us; skew 500 ppm
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:26.680639 2641 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:26.680987 2641 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:26.681072 2641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:26.681232 2641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:26.682364 2641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "0ec27269c7914d5e99e17780823aaf24"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "03ff1f3713651cccce4da21755079ce0"
server_key_iv: "a12f3384306ab924a207834fc8ae81b2"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.682734 2641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "0ec27269c7914d5e99e17780823aaf24"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "03ff1f3713651cccce4da21755079ce0"
server_key_iv: "a12f3384306ab924a207834fc8ae81b2"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.684231 2641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:26.685050 2655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.685251 2641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:26.685336 2641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "0ec27269c7914d5e99e17780823aaf24"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "03ff1f3713651cccce4da21755079ce0"
server_key_iv: "a12f3384306ab924a207834fc8ae81b2"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.685392 2641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:26.697374 2641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:26.699501 2641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:26.699622 2641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:26.699801 2641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:26.700099 2641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:26.700143 2641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.700166 2641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:26.700208 2641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.706349 2641 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:46183
I20260806 03:42:26.706414 2776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:46183 every 8 connection(s)
I20260806 03:42:26.706715 2641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:26.709353 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2641
I20260806 03:42:26.709471 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:26.709666 30816 external_mini_cluster.cc:1428] Setting key 29d5351d394f36e6e467883d7f2db6ca
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:26.717056 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:26.707479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:44699 (local address 127.30.24.62:36171)
0806 03:42:26.707541 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:26.707543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:26.707769 (+ 226us) server_negotiation.cc:407] Connection header received
0806 03:42:26.708271 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:26.708272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:26.708284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:26.708339 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:26.709389 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.710053 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.711229 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.711547 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.711622 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:26.714109 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:26.714111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:26.714111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:26.714124 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:26.715623 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:26.716024 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:26.716025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:26.716026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:26.716092 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:26.716344 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:26.716345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:26.716345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:26.716516 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:26.716582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:26.716863 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 03:42:26.716962 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:42:26.717234 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:26.707429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36171 (local address 127.30.24.1:44699)
0806 03:42:26.707628 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:26.707672 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:26.708180 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:26.708400 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:26.708404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:26.708740 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:26.709285 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:26.709291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.710169 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.710170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:26.711134 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:26.711137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.711634 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.711635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:26.711739 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:26.712339 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:26.712350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:26.713991 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:26.715711 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:26.715719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:26.715722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:26.715944 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:26.716168 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:26.716169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:26.716170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:26.716276 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:26.716572 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:26.716580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:26.716719 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 03:42:26.716825 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 03:42:26.716987 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260806 03:42:26.717798 2777 heartbeater.cc:347] Connected to a master server at 127.30.24.62:36171
I20260806 03:42:26.717919 2777 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:26.718094 2777 heartbeater.cc:511] Master 127.30.24.62:36171 requested a full tablet report, sending...
I20260806 03:42:26.718647 2578 ts_manager.cc:194] Registered new tserver with Master: 0ec27269c7914d5e99e17780823aaf24 (127.30.24.1:46183)
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
I20260806 03:42:26.719256 2578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:44699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:26.737166 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36171
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:26.821336 2782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:26.821540 2782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:26.821563 2782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:26.822749 2782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:26.822791 2782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:26.822817 2782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:26.824306 2782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:26.824590 2782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:26.824827 2782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:26.829912 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.829957 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.829931 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:26.830201 2782 server_base.cc:1034] running on GCE node
I20260806 03:42:26.830454 2782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:26.830722 2782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:26.831907 2782 hybrid_clock.cc:648] HybridClock initialized: now 1785987746831904 us; error 39 us; skew 500 ppm
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:26.834605 2782 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:26.835035 2782 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:26.835129 2782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:26.835279 2782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:26.836650 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "6ab0454228bf463e8699ffe0ee72016e"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "3826f7278bc6195aa86549210eda162d"
server_key_iv: "979c8e7cb7db841e2925eb66db48bc0b"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.837026 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "6ab0454228bf463e8699ffe0ee72016e"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "3826f7278bc6195aa86549210eda162d"
server_key_iv: "979c8e7cb7db841e2925eb66db48bc0b"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.838392 2782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:26.839205 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.839397 2782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:26.839510 2782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "6ab0454228bf463e8699ffe0ee72016e"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "3826f7278bc6195aa86549210eda162d"
server_key_iv: "979c8e7cb7db841e2925eb66db48bc0b"
server_key_version: "encryptionkey@0"
I20260806 03:42:26.839591 2782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:26.857127 2782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:26.859527 2782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:26.859699 2782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:26.859941 2782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:26.860276 2782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:26.860334 2782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.860376 2782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:26.860404 2782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:26.866671 2782 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:33625
I20260806 03:42:26.866714 2913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:33625 every 8 connection(s)
I20260806 03:42:26.866989 2782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:26.872983 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2782
I20260806 03:42:26.873119 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:26.873359 30816 external_mini_cluster.cc:1428] Setting key 120cdd0da1ec3370824f630b24f03c07
I20260806 03:42:26.877100 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:26.867731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:53311 (local address 127.30.24.62:36171)
0806 03:42:26.867798 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:26.867801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:26.868044 (+ 243us) server_negotiation.cc:407] Connection header received
0806 03:42:26.868590 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:26.868591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:26.868601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:26.868660 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:26.869578 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.870079 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.871116 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.871317 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.871373 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:26.873721 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:26.873722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:26.873723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:26.873731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:26.875398 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:26.875970 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:26.875972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:26.875972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:26.876039 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:26.876340 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:26.876340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:26.876341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:26.876511 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:26.876579 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:26.876948 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 03:42:26.877025 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:42:26.877350 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:26.867707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36171 (local address 127.30.24.2:53311)
0806 03:42:26.867902 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:26.867945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:26.868468 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:26.868712 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:26.868715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:26.868964 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:26.869477 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:26.869481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.870165 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.870166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:26.871029 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:26.871031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:26.871395 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:26.871396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:26.871491 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:26.871981 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:26.871989 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:26.873557 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:26.875500 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:26.875508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:26.875510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:26.875871 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:26.876121 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:26.876122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:26.876122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:26.876270 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:26.876593 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:26.876599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:26.876775 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 03:42:26.876897 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 03:42:26.877099 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260806 03:42:26.877836 2914 heartbeater.cc:347] Connected to a master server at 127.30.24.62:36171
I20260806 03:42:26.877954 2914 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:26.878123 2914 heartbeater.cc:511] Master 127.30.24.62:36171 requested a full tablet report, sending...
I20260806 03:42:26.878533 2578 ts_manager.cc:194] Registered new tserver with Master: 6ab0454228bf463e8699ffe0ee72016e (127.30.24.2:33625)
I20260806 03:42:26.879070 2578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:53311
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:26.901430 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36171
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:26.985275 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:26.985486 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:26.985510 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:26.986641 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:26.986680 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:26.986706 2919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:26.988251 2919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:36171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:26.988569 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:26.988839 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:26.993906 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.993906 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:26.993894 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:26.994108 2919 server_base.cc:1034] running on GCE node
I20260806 03:42:26.994395 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:26.994663 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:26.995836 2919 hybrid_clock.cc:648] HybridClock initialized: now 1785987746995800 us; error 55 us; skew 500 ppm
Aug 06 03:42:26 dist-test-slave-10pc krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:26.998473 2919 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:26.998835 2919 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:26.998911 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:26.999068 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:27.000161 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "71f7c7f9ab6847eca41cabbe1d17b467"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "6e5f2929ab25b821aa4d4b1bb1a9e0b4"
server_key_iv: "0283d1e05f4942c5ff3273c519e64e35"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.000602 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "71f7c7f9ab6847eca41cabbe1d17b467"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "6e5f2929ab25b821aa4d4b1bb1a9e0b4"
server_key_iv: "0283d1e05f4942c5ff3273c519e64e35"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.001941 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:27.002817 2933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.003018 2919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:27.003108 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "71f7c7f9ab6847eca41cabbe1d17b467"
format_stamp: "Formatted at 2026-08-06 03:42:26 on dist-test-slave-10pc"
server_key: "6e5f2929ab25b821aa4d4b1bb1a9e0b4"
server_key_iv: "0283d1e05f4942c5ff3273c519e64e35"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.003191 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:27.039611 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:27.041810 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:27.041970 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:27.042246 2919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:27.042644 2919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:27.042706 2919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.042752 2919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:27.042778 2919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.049352 2919 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:34741
I20260806 03:42:27.049403 3048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:34741 every 8 connection(s)
I20260806 03:42:27.049691 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:42:27 dist-test-slave-10pc krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:27.057433 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 2919
I20260806 03:42:27.057541 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:27.057776 30816 external_mini_cluster.cc:1428] Setting key 44750303810f920b806761319b83ca9e
I20260806 03:42:27.059458 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.050467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:34789 (local address 127.30.24.62:36171)
0806 03:42:27.050539 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.050542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.050805 (+ 263us) server_negotiation.cc:407] Connection header received
0806 03:42:27.051332 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.051333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.051344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.051397 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.052342 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.052897 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.053925 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.054095 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.054149 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.056508 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.056510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.056510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:27.056518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.058028 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:27.058509 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:27.058511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:27.058511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:27.058581 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:27.058883 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:27.058884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:27.058884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:27.058997 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.059051 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.059321 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.059394 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 03:42:27.059612 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.050461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36171 (local address 127.30.24.3:34789)
0806 03:42:27.050670 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.050711 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.051242 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.051454 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.051456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.051724 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.052263 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.052269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.052986 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.052987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.053850 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.053852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.054189 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.054190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.054326 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.054915 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:27.054924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.056376 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.058090 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:27.058098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:27.058100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:27.058425 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:27.058666 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:27.058667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:27.058667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:27.058811 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:27.059057 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.059064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.059193 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.059281 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.059405 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260806 03:42:27.060077 3050 heartbeater.cc:347] Connected to a master server at 127.30.24.62:36171
I20260806 03:42:27.060194 3050 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:27.060364 3050 heartbeater.cc:511] Master 127.30.24.62:36171 requested a full tablet report, sending...
I20260806 03:42:27.060786 2578 ts_manager.cc:194] Registered new tserver with Master: 71f7c7f9ab6847eca41cabbe1d17b467 (127.30.24.3:34741)
I20260806 03:42:27.061172 2578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:34789
I20260806 03:42:27.069675 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:42:27.077551 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.071002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.30.24.62:36171)
0806 03:42:27.071150 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.071152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.071243 (+ 91us) server_negotiation.cc:407] Connection header received
0806 03:42:27.071288 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.071288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.071297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.071344 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.071891 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.072366 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.073199 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.073396 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.073456 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.074773 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.074775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.074775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:27.074782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.076262 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:27.076780 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:27.076781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:27.076781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:27.076848 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:27.077012 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:27.077013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:27.077013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:27.077166 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.077238 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.077394 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.077456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 03:42:27.079391 2578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:27.081147 2578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:27.087990 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.083829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:34741 (local address 127.0.0.1:41816)
0806 03:42:27.084241 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.084251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.084306 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.084676 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.084677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.084679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:27.084726 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.084728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.085609 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.085610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.086890 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.086892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.087692 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.087693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.087750 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.087760 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.087846 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.087906 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":5,"thread_start_us":76,"threads_started":1}
I20260806 03:42:27.088096 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.083948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.30.24.1:46183)
0806 03:42:27.084147 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.084153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.084285 (+ 132us) server_negotiation.cc:407] Connection header received
0806 03:42:27.084330 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.084333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.084348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.084742 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:27.084838 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.085894 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.086973 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.087727 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.087811 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.087875 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.087951 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.087998 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260806 03:42:27.088328 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.083600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:33625 (local address 127.0.0.1:35290)
0806 03:42:27.083812 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.083856 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.083921 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.084355 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.084358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.084361 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:27.084442 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.084444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.085904 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.085905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.086968 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.086970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.087860 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.087861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.087940 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.087945 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.087977 (+ 32us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.088018 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260806 03:42:27.088376 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.083656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:46183 (local address 127.0.0.1:48826)
0806 03:42:27.083813 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.083857 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.083923 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.084443 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.084444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.084445 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:27.084505 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.084507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.085972 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.085973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.086880 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.086883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.087818 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.087819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.087875 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.087878 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.087935 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.087967 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260806 03:42:27.088609 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.084038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.30.24.3:34741)
0806 03:42:27.084235 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.084240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.084364 (+ 124us) server_negotiation.cc:407] Connection header received
0806 03:42:27.084408 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.084410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.084427 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.084518 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:27.084817 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.085499 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.086994 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.087510 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.087599 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.087660 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.088410 (+ 750us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.088501 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260806 03:42:27.088783 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.083737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.30.24.2:33625)
0806 03:42:27.084164 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.084168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.084179 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:27.084221 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.084223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.084240 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.084547 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:27.084654 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.085798 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.087035 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.087765 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.088483 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.088566 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.088594 (+ 28us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.088680 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":88,"threads_started":1}
I20260806 03:42:27.089376 2982 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad5d077f45e47fa83c2d2f647d72d0c (DEFAULT_TABLE table=test-table [id=8692e05ff4ba47efbf71fdd4b59041db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:27.089419 2846 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad5d077f45e47fa83c2d2f647d72d0c (DEFAULT_TABLE table=test-table [id=8692e05ff4ba47efbf71fdd4b59041db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:27.089419 2707 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad5d077f45e47fa83c2d2f647d72d0c (DEFAULT_TABLE table=test-table [id=8692e05ff4ba47efbf71fdd4b59041db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:27.089749 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad5d077f45e47fa83c2d2f647d72d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.089752 2846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad5d077f45e47fa83c2d2f647d72d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.089752 2707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad5d077f45e47fa83c2d2f647d72d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.092339 3067 tablet_bootstrap.cc:492] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Bootstrap starting.
I20260806 03:42:27.093102 3067 tablet_bootstrap.cc:654] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.093209 3068 tablet_bootstrap.cc:492] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Bootstrap starting.
I20260806 03:42:27.093461 3069 tablet_bootstrap.cc:492] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: Bootstrap starting.
I20260806 03:42:27.093773 3067 log.cc:824] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.093909 3068 tablet_bootstrap.cc:654] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.094165 3069 tablet_bootstrap.cc:654] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.094480 3068 log.cc:824] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.094717 3067 tablet_bootstrap.cc:492] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: No bootstrap required, opened a new log
I20260806 03:42:27.094794 3067 ts_tablet_manager.cc:1397] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:42:27.094785 3069 log.cc:824] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.095410 3068 tablet_bootstrap.cc:492] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: No bootstrap required, opened a new log
I20260806 03:42:27.095476 3069 tablet_bootstrap.cc:492] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: No bootstrap required, opened a new log
I20260806 03:42:27.095549 3069 ts_tablet_manager.cc:1397] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:27.095579 3068 ts_tablet_manager.cc:1397] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:27.096935 3067 raft_consensus.cc:348] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.097122 3067 raft_consensus.cc:374] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.097021 3069 raft_consensus.cc:348] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.097163 3067 raft_consensus.cc:729] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f7c7f9ab6847eca41cabbe1d17b467, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.097198 3069 raft_consensus.cc:374] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.097234 3069 raft_consensus.cc:729] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec27269c7914d5e99e17780823aaf24, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.097298 3067 consensus_queue.cc:260] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.097373 3069 consensus_queue.cc:260] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.097609 3067 ts_tablet_manager.cc:1428] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:27.097690 3069 ts_tablet_manager.cc:1428] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:27.097738 3050 heartbeater.cc:503] Master 127.30.24.62:36171 was elected leader, sending a full tablet report...
I20260806 03:42:27.097749 2777 heartbeater.cc:503] Master 127.30.24.62:36171 was elected leader, sending a full tablet report...
I20260806 03:42:27.097790 3068 raft_consensus.cc:348] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.097927 3068 raft_consensus.cc:374] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.097962 3068 raft_consensus.cc:729] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab0454228bf463e8699ffe0ee72016e, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.098097 3068 consensus_queue.cc:260] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.098389 3068 ts_tablet_manager.cc:1428] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:27.098506 2914 heartbeater.cc:503] Master 127.30.24.62:36171 was elected leader, sending a full tablet report...
W20260806 03:42:27.117537 2915 tablet.cc:2369] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:42:27.207501 2778 tablet.cc:2369] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:27.281992 3075 raft_consensus.cc:482] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.282172 3075 raft_consensus.cc:504] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.282614 3075 leader_election.cc:288] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71f7c7f9ab6847eca41cabbe1d17b467 (127.30.24.3:34741), 0ec27269c7914d5e99e17780823aaf24 (127.30.24.1:46183)
I20260806 03:42:27.286055 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.282928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:40377 (local address 127.30.24.1:46183)
0806 03:42:27.282978 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.282981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.283034 (+ 53us) server_negotiation.cc:407] Connection header received
0806 03:42:27.283069 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.283070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.283083 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.283145 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:27.283352 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.283885 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.285308 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.285653 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.285709 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.285718 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.285933 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.285978 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:42:27.286092 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.282837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:34741 (local address 127.30.24.2:50147)
0806 03:42:27.283081 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.283092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.283172 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.283345 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.283346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.283348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:27.283416 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.283418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.284226 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.284227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.285175 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.285178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.285813 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.285814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.285872 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.285896 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.285953 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.285989 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":43,"thread_start_us":135,"threads_started":1}
I20260806 03:42:27.286104 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.282936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:50147 (local address 127.30.24.3:34741)
0806 03:42:27.282979 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.282981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.283182 (+ 201us) server_negotiation.cc:407] Connection header received
0806 03:42:27.283221 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.283222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.283245 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.283311 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:27.283525 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.284130 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.285262 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.285726 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.285804 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.285816 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.285976 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.286027 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:42:27.286098 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.282837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:46183 (local address 127.30.24.2:40377)
0806 03:42:27.282912 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.282925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.282976 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.283186 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.283187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.283191 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:27.283271 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.283274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.283979 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.283981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.285185 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:27.285206 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.285736 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.285737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:27.285838 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.285843 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.285901 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.285961 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":60}
I20260806 03:42:27.286538 3003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad5d077f45e47fa83c2d2f647d72d0c" candidate_uuid: "6ab0454228bf463e8699ffe0ee72016e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" is_pre_election: true
I20260806 03:42:27.286747 3003 raft_consensus.cc:2454] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab0454228bf463e8699ffe0ee72016e in term 0.
I20260806 03:42:27.286734 2728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad5d077f45e47fa83c2d2f647d72d0c" candidate_uuid: "6ab0454228bf463e8699ffe0ee72016e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec27269c7914d5e99e17780823aaf24" is_pre_election: true
I20260806 03:42:27.286935 2728 raft_consensus.cc:2454] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab0454228bf463e8699ffe0ee72016e in term 0.
I20260806 03:42:27.286993 2797 leader_election.cc:302] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ab0454228bf463e8699ffe0ee72016e, 71f7c7f9ab6847eca41cabbe1d17b467; no voters:
I20260806 03:42:27.287159 3075 raft_consensus.cc:2781] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:27.287237 3075 raft_consensus.cc:482] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:27.287266 3075 raft_consensus.cc:3037] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.288079 3075 raft_consensus.cc:504] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.288257 3075 leader_election.cc:288] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [CANDIDATE]: Term 1 election: Requested vote from peers 71f7c7f9ab6847eca41cabbe1d17b467 (127.30.24.3:34741), 0ec27269c7914d5e99e17780823aaf24 (127.30.24.1:46183)
I20260806 03:42:27.288481 2728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad5d077f45e47fa83c2d2f647d72d0c" candidate_uuid: "6ab0454228bf463e8699ffe0ee72016e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec27269c7914d5e99e17780823aaf24"
I20260806 03:42:27.288564 3003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad5d077f45e47fa83c2d2f647d72d0c" candidate_uuid: "6ab0454228bf463e8699ffe0ee72016e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f7c7f9ab6847eca41cabbe1d17b467"
I20260806 03:42:27.288579 2728 raft_consensus.cc:3037] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.288652 3003 raft_consensus.cc:3037] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.289547 3003 raft_consensus.cc:2454] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab0454228bf463e8699ffe0ee72016e in term 1.
I20260806 03:42:27.289547 2728 raft_consensus.cc:2454] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab0454228bf463e8699ffe0ee72016e in term 1.
I20260806 03:42:27.289752 2797 leader_election.cc:302] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ab0454228bf463e8699ffe0ee72016e, 71f7c7f9ab6847eca41cabbe1d17b467; no voters:
I20260806 03:42:27.289848 3075 raft_consensus.cc:2781] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:27.290000 3075 raft_consensus.cc:686] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 1 LEADER]: Becoming Leader. State: Replica: 6ab0454228bf463e8699ffe0ee72016e, State: Running, Role: LEADER
I20260806 03:42:27.290110 3075 consensus_queue.cc:237] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } }
I20260806 03:42:27.291070 2575 catalog_manager.cc:5495] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ab0454228bf463e8699ffe0ee72016e (127.30.24.2). New cstate: current_term: 1 leader_uuid: "6ab0454228bf463e8699ffe0ee72016e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0454228bf463e8699ffe0ee72016e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 33625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 } health_report { overall_health: UNKNOWN } } }
W20260806 03:42:27.300276 3051 tablet.cc:2369] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:27.318778 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.316187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.30.24.2:33625)
0806 03:42:27.316259 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.316262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.316271 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:27.316324 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.316325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.316347 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.316431 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:27.316615 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.317111 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.317911 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:27.318114 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:27.318195 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:27.318392 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:27.318509 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:27.318616 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.318642 (+ 26us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.318665 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 03:42:27.320706 2728 raft_consensus.cc:1261] T 9ad5d077f45e47fa83c2d2f647d72d0c P 0ec27269c7914d5e99e17780823aaf24 [term 1 FOLLOWER]: Refusing update from remote peer 6ab0454228bf463e8699ffe0ee72016e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:27.320705 3003 raft_consensus.cc:1261] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 1 FOLLOWER]: Refusing update from remote peer 6ab0454228bf463e8699ffe0ee72016e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:27.321099 3075 consensus_queue.cc:1035] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec27269c7914d5e99e17780823aaf24" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:27.321296 3077 consensus_queue.cc:1035] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f7c7f9ab6847eca41cabbe1d17b467" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:27.324014 3081 mvcc.cc:205] Tried to move back new op lower bound from 7315405813022621696 to 7315405812901040128. Current Snapshot: MvccSnapshot[applied={T|T < 7315405813022621696}]
I20260806 03:42:27.326300 2575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:27.326406 2575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:27.327240 2575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a94d2d98e1734fa09c6aff742fbc0192: Sending DeleteTablet for 3 replicas of tablet 9ad5d077f45e47fa83c2d2f647d72d0c
I20260806 03:42:27.327637 2846 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad5d077f45e47fa83c2d2f647d72d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:27 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:35290
I20260806 03:42:27.327682 2707 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad5d077f45e47fa83c2d2f647d72d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:27 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:48826
I20260806 03:42:27.327821 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2641
I20260806 03:42:27.327874 2982 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad5d077f45e47fa83c2d2f647d72d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:27 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:41816
I20260806 03:42:27.329077 3090 tablet_replica.cc:330] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: stopping tablet replica
I20260806 03:42:27.329185 3090 raft_consensus.cc:2229] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:27.329282 3090 raft_consensus.cc:2258] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:27.334028 3092 tablet_replica.cc:330] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: stopping tablet replica
I20260806 03:42:27.334170 3092 raft_consensus.cc:2229] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:27.334297 3092 raft_consensus.cc:2258] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 03:42:27.334523 2561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:46183 (error 108)
I20260806 03:42:27.334831 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2782
I20260806 03:42:27.334872 3092 ts_tablet_manager.cc:1905] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:27.335086 3090 ts_tablet_manager.cc:1905] T 9ad5d077f45e47fa83c2d2f647d72d0c P 6ab0454228bf463e8699ffe0ee72016e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:27.336994 3092 ts_tablet_manager.cc:1918] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:27.337054 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.336914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:46183 (local address 127.0.0.1:48836)
0806 03:42:27.336971 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.337016 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:46183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 03:42:27.337076 3092 log.cc:1196] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/9ad5d077f45e47fa83c2d2f647d72d0c
W20260806 03:42:27.337180 2561 catalog_manager.cc:4584] TS 0ec27269c7914d5e99e17780823aaf24 (127.30.24.1:46183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ad5d077f45e47fa83c2d2f647d72d0c: Network error: Client connection negotiation failed: client connection to 127.30.24.1:46183: connect: Connection refused (error 111)
I20260806 03:42:27.337390 3092 ts_tablet_manager.cc:1939] T 9ad5d077f45e47fa83c2d2f647d72d0c P 71f7c7f9ab6847eca41cabbe1d17b467: Deleting consensus metadata
I20260806 03:42:27.337843 2560 catalog_manager.cc:4841] TS 71f7c7f9ab6847eca41cabbe1d17b467 (127.30.24.3:34741): tablet 9ad5d077f45e47fa83c2d2f647d72d0c (table test-table [id=8692e05ff4ba47efbf71fdd4b59041db]) successfully deleted
I20260806 03:42:27.340793 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2919
I20260806 03:42:27.340857 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.340753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:33625 (local address 127.0.0.1:35296)
0806 03:42:27.340795 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.340822 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.2:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 03:42:27.346709 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 2544
2026-08-06T03:42:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2175 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T03:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:27Z Disabled control of system clock
I20260806 03:42:27.376571 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:43759
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:43759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 03:42:27.465652 3097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:27.465863 3097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:27.465885 3097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:27.467096 3097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:27.467146 3097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:27.467192 3097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:27.467213 3097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:27.467242 3097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:27.468668 3097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:43759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:43759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:27.468883 3097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:27.469156 3097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:27.474139 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.474165 3102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.474139 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:27.474469 3097 server_base.cc:1034] running on GCE node
I20260806 03:42:27.474763 3097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:27.475064 3097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:27.476238 3097 hybrid_clock.cc:648] HybridClock initialized: now 1785987747476215 us; error 38 us; skew 500 ppm
I20260806 03:42:27.477519 3097 webserver.cc:460] Webserver started at http://127.30.24.62:36287/ using document root <none> and password file <none>
I20260806 03:42:27.477795 3097 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:27.477867 3097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:27.478024 3097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:27.479162 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "5096a9180bf944f2b09b5daa98ba7521"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "30bbfbddcd97481b0e123cef243b16e9"
server_key_iv: "67c4435ba86ffce1007f2ea27eafc2b6"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.479537 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "5096a9180bf944f2b09b5daa98ba7521"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "30bbfbddcd97481b0e123cef243b16e9"
server_key_iv: "67c4435ba86ffce1007f2ea27eafc2b6"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.481030 3097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:27.481861 3111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.482051 3097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:27.482177 3097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "5096a9180bf944f2b09b5daa98ba7521"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "30bbfbddcd97481b0e123cef243b16e9"
server_key_iv: "67c4435ba86ffce1007f2ea27eafc2b6"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.482256 3097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:27.485662 3097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:27.485935 3097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:27.486078 3097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:27.489950 3097 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:43759
I20260806 03:42:27.489982 3164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:43759 every 8 connection(s)
I20260806 03:42:27.490336 3097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:27.491026 3165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.491489 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3097
I20260806 03:42:27.491628 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:27.491823 30816 external_mini_cluster.cc:1428] Setting key 1a91d1f7e7bd6231243816c50e113cc3
I20260806 03:42:27.494807 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.492506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.30.24.62:43759)
0806 03:42:27.492745 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.492755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.492790 (+ 35us) server_negotiation.cc:407] Connection header received
0806 03:42:27.493325 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.493334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.493390 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.493546 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.494340 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.494345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.494345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.494349 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.494400 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.494414 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.494491 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.494551 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.494614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260806 03:42:27.495234 3165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521: Bootstrap starting.
I20260806 03:42:27.495923 3165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.496230 3165 log.cc:824] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.496954 3165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521: No bootstrap required, opened a new log
I20260806 03:42:27.498324 3165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096a9180bf944f2b09b5daa98ba7521" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 43759 } }
I20260806 03:42:27.498488 3165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.498541 3165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5096a9180bf944f2b09b5daa98ba7521, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.498668 3165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096a9180bf944f2b09b5daa98ba7521" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 43759 } }
I20260806 03:42:27.498764 3165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:27.498801 3165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:27.498852 3165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.499394 3165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096a9180bf944f2b09b5daa98ba7521" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 43759 } }
I20260806 03:42:27.499516 3165 leader_election.cc:302] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5096a9180bf944f2b09b5daa98ba7521; no voters:
I20260806 03:42:27.499693 3165 leader_election.cc:288] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:27.500034 3165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:27.500059 3170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:27.500196 3170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [term 1 LEADER]: Becoming Leader. State: Replica: 5096a9180bf944f2b09b5daa98ba7521, State: Running, Role: LEADER
I20260806 03:42:27.500293 3170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096a9180bf944f2b09b5daa98ba7521" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 43759 } }
I20260806 03:42:27.500685 3170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5096a9180bf944f2b09b5daa98ba7521. Latest consensus state: current_term: 1 leader_uuid: "5096a9180bf944f2b09b5daa98ba7521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096a9180bf944f2b09b5daa98ba7521" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 43759 } } }
I20260806 03:42:27.500771 3170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:27.500720 3171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5096a9180bf944f2b09b5daa98ba7521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5096a9180bf944f2b09b5daa98ba7521" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 43759 } } }
I20260806 03:42:27.501003 3171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:27.501434 3180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:27.501909 3180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:27.503415 3180 catalog_manager.cc:1347] Generated new cluster ID: 4fa0a31034704de3b30c7ffa31522e76
I20260806 03:42:27.503477 3180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:27.508646 3180 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:27.508822 3180 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:27.521896 3180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521: Generated new TSK 0
I20260806 03:42:27.522145 3180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:27.528867 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:43759
--builtin_ntp_servers=127.30.24.20:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 03:42:27.613891 3190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:27.614147 3190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:27.614185 3190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:27.615379 3190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:27.615450 3190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:27.615494 3190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:27.616981 3190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:43759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:27.617273 3190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:27.617597 3190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:27.623689 3197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:27.623927 3190 server_base.cc:1034] running on GCE node
W20260806 03:42:27.623699 3199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.623699 3196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:27.624282 3190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:27.624655 3190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:27.625849 3190 hybrid_clock.cc:648] HybridClock initialized: now 1785987747625835 us; error 30 us; skew 500 ppm
I20260806 03:42:27.627295 3190 webserver.cc:460] Webserver started at http://127.30.24.1:35917/ using document root <none> and password file <none>
I20260806 03:42:27.627592 3190 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:27.627671 3190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:27.627830 3190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:27.629119 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "95c79c9638f14eb7b46530b5eb77ddf3"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "0043d2f00132360ca4c0f4dc70f0ed25"
server_key_iv: "4e40a45ea73278a74a1d208d8c7a25aa"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.629509 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "95c79c9638f14eb7b46530b5eb77ddf3"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "0043d2f00132360ca4c0f4dc70f0ed25"
server_key_iv: "4e40a45ea73278a74a1d208d8c7a25aa"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.630889 3190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:27.631731 3205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.631963 3190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:27.632081 3190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "95c79c9638f14eb7b46530b5eb77ddf3"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "0043d2f00132360ca4c0f4dc70f0ed25"
server_key_iv: "4e40a45ea73278a74a1d208d8c7a25aa"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.632164 3190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:27.635783 3190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:27.636072 3190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:27.636225 3190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:27.636440 3190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:27.636821 3190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:27.636878 3190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.636917 3190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:27.636946 3190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.642798 3190 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:44923
I20260806 03:42:27.642838 3323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:44923 every 8 connection(s)
I20260806 03:42:27.643191 3190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:27.644809 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3190
I20260806 03:42:27.644938 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:27.645129 30816 external_mini_cluster.cc:1428] Setting key 2a69f8da2b181c268eeadef65adac70f
I20260806 03:42:27.646304 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:43759
--builtin_ntp_servers=127.30.24.20:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 03:42:27.646726 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.643951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:49889 (local address 127.30.24.62:43759)
0806 03:42:27.644039 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.644042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.644246 (+ 204us) server_negotiation.cc:407] Connection header received
0806 03:42:27.644849 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.644850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.644857 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.644911 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.646412 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.646413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.646413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.646419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.646437 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.646444 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.646506 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.646657 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.646673 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 03:42:27.646965 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.643913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:43759 (local address 127.30.24.1:49889)
0806 03:42:27.644118 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.644152 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.644736 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.645011 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.645015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.645057 (+ 42us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.646236 (+ 1179us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.646236 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.646267 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.646297 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.646538 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.646546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.646549 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.646647 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.646715 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260806 03:42:27.647500 3324 heartbeater.cc:347] Connected to a master server at 127.30.24.62:43759
I20260806 03:42:27.647611 3324 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:27.647774 3324 heartbeater.cc:511] Master 127.30.24.62:43759 requested a full tablet report, sending...
I20260806 03:42:27.648447 3128 ts_manager.cc:194] Registered new tserver with Master: 95c79c9638f14eb7b46530b5eb77ddf3 (127.30.24.1:44923)
W20260806 03:42:27.731155 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:27.731354 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:27.731376 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:27.732653 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:27.732695 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:27.732721 3327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:27.733992 3327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:43759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:27.734182 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:27.734421 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:27.739703 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.739703 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.739733 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:27.740091 3327 server_base.cc:1034] running on GCE node
I20260806 03:42:27.740286 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:27.740542 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:27.741714 3327 hybrid_clock.cc:648] HybridClock initialized: now 1785987747741708 us; error 33 us; skew 500 ppm
I20260806 03:42:27.743065 3327 webserver.cc:460] Webserver started at http://127.30.24.2:38159/ using document root <none> and password file <none>
I20260806 03:42:27.743338 3327 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:27.743396 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:27.743494 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:27.744642 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "0988695b157b46ffbd950a438210758e"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "aeba03a018481d182aaa01b0dc2613a6"
server_key_iv: "e3dd617a97109ab4f8cde5ec849916ad"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.744961 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "0988695b157b46ffbd950a438210758e"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "aeba03a018481d182aaa01b0dc2613a6"
server_key_iv: "e3dd617a97109ab4f8cde5ec849916ad"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.746230 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:27.746902 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.747078 3327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:27.747157 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "0988695b157b46ffbd950a438210758e"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "aeba03a018481d182aaa01b0dc2613a6"
server_key_iv: "e3dd617a97109ab4f8cde5ec849916ad"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.747212 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:27.750479 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:27.750700 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:27.750792 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:27.750946 3327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:27.751224 3327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:27.751257 3327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.751276 3327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:27.751289 3327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.756850 3327 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:45519
I20260806 03:42:27.756990 3459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:45519 every 8 connection(s)
I20260806 03:42:27.757186 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:27.760107 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.757949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:45593 (local address 127.30.24.62:43759)
0806 03:42:27.758070 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.758072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.758247 (+ 175us) server_negotiation.cc:407] Connection header received
0806 03:42:27.758694 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.758695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.758700 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.758752 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.759920 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.759921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.759921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.759925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.759939 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.759943 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.759990 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.760043 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.760054 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 03:42:27.760301 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.757912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:43759 (local address 127.30.24.2:45593)
0806 03:42:27.758108 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.758161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.758606 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.758802 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.758805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.758813 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.759780 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.759781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.759808 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.759836 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.759989 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.759991 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.759993 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.760043 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.760089 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260806 03:42:27.760877 3460 heartbeater.cc:347] Connected to a master server at 127.30.24.62:43759
I20260806 03:42:27.760957 3460 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:27.761121 3460 heartbeater.cc:511] Master 127.30.24.62:43759 requested a full tablet report, sending...
I20260806 03:42:27.761529 3128 ts_manager.cc:194] Registered new tserver with Master: 0988695b157b46ffbd950a438210758e (127.30.24.2:45519)
I20260806 03:42:27.762264 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3327
I20260806 03:42:27.762377 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:27.762564 30816 external_mini_cluster.cc:1428] Setting key 8490298a3262373200802b9af60c398c
I20260806 03:42:27.763440 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:43759
--builtin_ntp_servers=127.30.24.20:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 03:42:27.846695 3463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:27.846894 3463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:27.846915 3463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:27.848071 3463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:27.848111 3463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:27.848137 3463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:27.849529 3463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:43759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:27.849844 3463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:27.850143 3463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:27.855167 3468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.855177 3469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:27.855211 3471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:27.855576 3463 server_base.cc:1034] running on GCE node
I20260806 03:42:27.855799 3463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:27.856046 3463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:27.857211 3463 hybrid_clock.cc:648] HybridClock initialized: now 1785987747857202 us; error 34 us; skew 500 ppm
I20260806 03:42:27.858655 3463 webserver.cc:460] Webserver started at http://127.30.24.3:42951/ using document root <none> and password file <none>
I20260806 03:42:27.858924 3463 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:27.858979 3463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:27.859182 3463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:27.860392 3463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "6cd6a9f8df344972bb85fccfab2ef578"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "4fe5cef880bf7d2efc254b197db52560"
server_key_iv: "22371eb06d40f7da324d8dc72bb41335"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.860877 3463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "6cd6a9f8df344972bb85fccfab2ef578"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "4fe5cef880bf7d2efc254b197db52560"
server_key_iv: "22371eb06d40f7da324d8dc72bb41335"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.862473 3463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:27.863320 3477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.863473 3463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:27.863566 3463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "6cd6a9f8df344972bb85fccfab2ef578"
format_stamp: "Formatted at 2026-08-06 03:42:27 on dist-test-slave-10pc"
server_key: "4fe5cef880bf7d2efc254b197db52560"
server_key_iv: "22371eb06d40f7da324d8dc72bb41335"
server_key_version: "encryptionkey@0"
I20260806 03:42:27.863622 3463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:27.866899 3463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:27.867110 3463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:27.867213 3463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:27.867367 3463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:27.867691 3463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:27.867725 3463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.867746 3463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:27.867787 3463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:27.873584 3463 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:44491
I20260806 03:42:27.873665 3594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:44491 every 8 connection(s)
I20260806 03:42:27.873978 3463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:27.876961 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.874762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:46641 (local address 127.30.24.62:43759)
0806 03:42:27.874830 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.874832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.875039 (+ 207us) server_negotiation.cc:407] Connection header received
0806 03:42:27.875541 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.875542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.875546 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.875605 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.876764 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.876766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.876766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.876770 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.876785 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.876788 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.876836 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.876897 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.876909 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:42:27.877135 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.874714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:43759 (local address 127.30.24.3:46641)
0806 03:42:27.874903 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.874947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.875451 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.875659 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.875662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.875670 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.876618 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.876619 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.876647 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.876675 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.876835 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.876837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.876839 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.876892 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.876938 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260806 03:42:27.878574 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3463
I20260806 03:42:27.878675 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:27.878845 30816 external_mini_cluster.cc:1428] Setting key 65cfe4d2aa955704d60f6133579f0f4a
I20260806 03:42:27.882990 3595 heartbeater.cc:347] Connected to a master server at 127.30.24.62:43759
I20260806 03:42:27.883116 3595 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:27.883304 3595 heartbeater.cc:511] Master 127.30.24.62:43759 requested a full tablet report, sending...
I20260806 03:42:27.883728 3128 ts_manager.cc:194] Registered new tserver with Master: 6cd6a9f8df344972bb85fccfab2ef578 (127.30.24.3:44491)
I20260806 03:42:27.894388 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:42:27.896899 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.895820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.30.24.62:43759)
0806 03:42:27.895881 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.895882 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.895963 (+ 81us) server_negotiation.cc:407] Connection header received
0806 03:42:27.896022 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.896023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.896027 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.896076 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.896727 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.896728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.896728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.896733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.896747 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.896751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.896800 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.896835 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.896847 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 03:42:27.898433 3128 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:27.900203 3128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:27.904161 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.902347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:45519 (local address 127.0.0.1:58910)
0806 03:42:27.902554 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.902598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.902668 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.903149 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.903157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.903164 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.903647 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.903648 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.903666 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.903692 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.903980 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.903987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.903990 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.904068 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.904083 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260806 03:42:27.904162 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.902439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.30.24.2:45519)
0806 03:42:27.902949 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.902952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.902962 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:27.902989 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.902991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.903005 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.903377 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.903864 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.903868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.903868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.903872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.903910 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.903925 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.903986 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.904077 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.904090 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":143,"threads_started":1}
I20260806 03:42:27.904553 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.902602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44923 (local address 127.0.0.1:40910)
0806 03:42:27.903097 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.903106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.903146 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.903596 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.903597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.903599 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.904129 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.904129 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.904135 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.904144 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.904407 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.904408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.904409 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.904475 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.904488 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":84,"threads_started":1}
I20260806 03:42:27.904744 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.902367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:44491 (local address 127.0.0.1:52496)
0806 03:42:27.902922 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.902933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.902990 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.903494 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.903495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.903499 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.903834 (+ 335us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.903834 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.903840 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.903851 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.904374 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.904375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.904375 (+ 0us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.904416 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.904428 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":111,"threads_started":1}
I20260806 03:42:27.904764 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.902716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.30.24.1:44923)
0806 03:42:27.902911 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.902916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.903205 (+ 289us) server_negotiation.cc:407] Connection header received
0806 03:42:27.903238 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.903240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.903250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.903323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.904236 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.904245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.904246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.904250 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.904285 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.904310 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.904378 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.904665 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.904684 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260806 03:42:27.904907 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.902722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.30.24.3:44491)
0806 03:42:27.902918 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.902922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.902987 (+ 65us) server_negotiation.cc:407] Connection header received
0806 03:42:27.903072 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.903076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.903087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.903163 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.903934 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.903943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.903944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.903949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.903987 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.904018 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.904078 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.904508 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.904526 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260806 03:42:27.904915 3393 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89de3244f34bec929113684854c5c4 (DEFAULT_TABLE table=test-table [id=e06e1cb951594402952502323cba788e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:27.905359 3393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f89de3244f34bec929113684854c5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.905530 3527 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89de3244f34bec929113684854c5c4 (DEFAULT_TABLE table=test-table [id=e06e1cb951594402952502323cba788e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:27.905893 3527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f89de3244f34bec929113684854c5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.908169 3609 tablet_bootstrap.cc:492] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Bootstrap starting.
I20260806 03:42:27.908614 3610 tablet_bootstrap.cc:492] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Bootstrap starting.
I20260806 03:42:27.908762 3255 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89de3244f34bec929113684854c5c4 (DEFAULT_TABLE table=test-table [id=e06e1cb951594402952502323cba788e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:27.909112 3609 tablet_bootstrap.cc:654] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.909162 3255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f89de3244f34bec929113684854c5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:27.909353 3610 tablet_bootstrap.cc:654] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.909447 3609 log.cc:824] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.909713 3610 log.cc:824] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.910279 3609 tablet_bootstrap.cc:492] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: No bootstrap required, opened a new log
I20260806 03:42:27.910395 3609 ts_tablet_manager.cc:1397] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:27.910501 3610 tablet_bootstrap.cc:492] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: No bootstrap required, opened a new log
I20260806 03:42:27.910580 3610 ts_tablet_manager.cc:1397] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:27.912639 3613 tablet_bootstrap.cc:492] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: Bootstrap starting.
I20260806 03:42:27.912578 3610 raft_consensus.cc:348] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.912765 3610 raft_consensus.cc:374] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.912807 3610 raft_consensus.cc:729] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd6a9f8df344972bb85fccfab2ef578, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.912940 3610 consensus_queue.cc:260] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.913251 3610 ts_tablet_manager.cc:1428] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:27.913352 3613 tablet_bootstrap.cc:654] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.913403 3595 heartbeater.cc:503] Master 127.30.24.62:43759 was elected leader, sending a full tablet report...
I20260806 03:42:27.913435 3609 raft_consensus.cc:348] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.913573 3609 raft_consensus.cc:374] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.913604 3609 raft_consensus.cc:729] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0988695b157b46ffbd950a438210758e, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.913658 3613 log.cc:824] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:27.913748 3609 consensus_queue.cc:260] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.913985 3609 ts_tablet_manager.cc:1428] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:27.914101 3460 heartbeater.cc:503] Master 127.30.24.62:43759 was elected leader, sending a full tablet report...
I20260806 03:42:27.914474 3613 tablet_bootstrap.cc:492] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: No bootstrap required, opened a new log
I20260806 03:42:27.914547 3613 ts_tablet_manager.cc:1397] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:27.916620 3613 raft_consensus.cc:348] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.916739 3613 raft_consensus.cc:374] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.916761 3613 raft_consensus.cc:729] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c79c9638f14eb7b46530b5eb77ddf3, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.916831 3613 consensus_queue.cc:260] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.917009 3613 ts_tablet_manager.cc:1428] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 03:42:27.917095 3324 heartbeater.cc:503] Master 127.30.24.62:43759 was elected leader, sending a full tablet report...
I20260806 03:42:27.955464 3614 raft_consensus.cc:482] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.955655 3614 raft_consensus.cc:504] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.956046 3614 leader_election.cc:288] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0988695b157b46ffbd950a438210758e (127.30.24.2:45519), 95c79c9638f14eb7b46530b5eb77ddf3 (127.30.24.1:44923)
I20260806 03:42:27.957630 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.956280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:45519 (local address 127.30.24.3:33871)
0806 03:42:27.956364 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.956379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.956442 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.956707 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.956708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.956712 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.957240 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.957240 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.957248 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.957278 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.957483 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.957484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.957485 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.957535 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.957549 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 03:42:27.957680 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.956367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:33871 (local address 127.30.24.2:45519)
0806 03:42:27.956563 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.956565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.956576 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:27.956609 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.956610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.956617 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.956685 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.957373 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.957374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.957375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.957379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.957397 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.957402 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.957470 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.957567 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.957585 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260806 03:42:27.957746 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.956566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52621 (local address 127.30.24.1:44923)
0806 03:42:27.956642 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.956644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.956713 (+ 69us) server_negotiation.cc:407] Connection header received
0806 03:42:27.956780 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.956781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.956787 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.956845 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.957521 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.957523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.957523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.957528 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.957545 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.957549 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.957596 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.957654 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.957668 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 03:42:27.957760 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.956427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44923 (local address 127.30.24.3:52621)
0806 03:42:27.956658 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.956668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:27.956720 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:27.956857 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:27.956858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:27.956879 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:27.957433 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:27.957434 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:27.957444 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:27.957461 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:27.957605 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:27.957606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:27.957607 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 03:42:27.957654 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 03:42:27.957666 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":151,"threads_started":1}
I20260806 03:42:27.957950 3413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89de3244f34bec929113684854c5c4" candidate_uuid: "6cd6a9f8df344972bb85fccfab2ef578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0988695b157b46ffbd950a438210758e" is_pre_election: true
I20260806 03:42:27.958108 3413 raft_consensus.cc:2454] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cd6a9f8df344972bb85fccfab2ef578 in term 0.
I20260806 03:42:27.958282 3277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89de3244f34bec929113684854c5c4" candidate_uuid: "6cd6a9f8df344972bb85fccfab2ef578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" is_pre_election: true
I20260806 03:42:27.958382 3480 leader_election.cc:302] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0988695b157b46ffbd950a438210758e, 6cd6a9f8df344972bb85fccfab2ef578; no voters:
I20260806 03:42:27.958459 3277 raft_consensus.cc:2454] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cd6a9f8df344972bb85fccfab2ef578 in term 0.
I20260806 03:42:27.958523 3614 raft_consensus.cc:2781] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:27.958596 3614 raft_consensus.cc:482] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:27.958619 3614 raft_consensus.cc:3037] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.959342 3614 raft_consensus.cc:504] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.959441 3614 leader_election.cc:288] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [CANDIDATE]: Term 1 election: Requested vote from peers 0988695b157b46ffbd950a438210758e (127.30.24.2:45519), 95c79c9638f14eb7b46530b5eb77ddf3 (127.30.24.1:44923)
I20260806 03:42:27.959653 3413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89de3244f34bec929113684854c5c4" candidate_uuid: "6cd6a9f8df344972bb85fccfab2ef578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0988695b157b46ffbd950a438210758e"
I20260806 03:42:27.959671 3277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89de3244f34bec929113684854c5c4" candidate_uuid: "6cd6a9f8df344972bb85fccfab2ef578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c79c9638f14eb7b46530b5eb77ddf3"
I20260806 03:42:27.959765 3413 raft_consensus.cc:3037] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.959764 3277 raft_consensus.cc:3037] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.960609 3413 raft_consensus.cc:2454] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cd6a9f8df344972bb85fccfab2ef578 in term 1.
I20260806 03:42:27.960659 3277 raft_consensus.cc:2454] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cd6a9f8df344972bb85fccfab2ef578 in term 1.
I20260806 03:42:27.960865 3480 leader_election.cc:302] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd6a9f8df344972bb85fccfab2ef578, 95c79c9638f14eb7b46530b5eb77ddf3; no voters:
I20260806 03:42:27.960980 3614 raft_consensus.cc:2781] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:27.961158 3614 raft_consensus.cc:686] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 1 LEADER]: Becoming Leader. State: Replica: 6cd6a9f8df344972bb85fccfab2ef578, State: Running, Role: LEADER
I20260806 03:42:27.961258 3614 consensus_queue.cc:237] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } }
I20260806 03:42:27.962096 3127 catalog_manager.cc:5495] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cd6a9f8df344972bb85fccfab2ef578 (127.30.24.3). New cstate: current_term: 1 leader_uuid: "6cd6a9f8df344972bb85fccfab2ef578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cd6a9f8df344972bb85fccfab2ef578" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44491 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:27.979657 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.978513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.30.24.3:44491)
0806 03:42:27.978603 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:27.978605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:27.978617 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:27.978672 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:27.978674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:27.978688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:27.978758 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:27.979406 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:27.979407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:27.979408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:27.979412 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:27.979429 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:27.979434 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:27.979506 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:27.979563 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 03:42:27.979577 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 03:42:27.981793 3277 raft_consensus.cc:1261] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 1 FOLLOWER]: Refusing update from remote peer 6cd6a9f8df344972bb85fccfab2ef578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:27.981757 3413 raft_consensus.cc:1261] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 1 FOLLOWER]: Refusing update from remote peer 6cd6a9f8df344972bb85fccfab2ef578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:27.982281 3619 consensus_queue.cc:1035] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95c79c9638f14eb7b46530b5eb77ddf3" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:27.982425 3614 consensus_queue.cc:1035] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0988695b157b46ffbd950a438210758e" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:27.984823 3622 mvcc.cc:205] Tried to move back new op lower bound from 7315405815730311168 to 7315405815650103296. Current Snapshot: MvccSnapshot[applied={T|T < 7315405815730311168}]
I20260806 03:42:27.985811 3623 mvcc.cc:205] Tried to move back new op lower bound from 7315405815730311168 to 7315405815650103296. Current Snapshot: MvccSnapshot[applied={T|T < 7315405815730311168}]
I20260806 03:42:27.987488 3127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:27.987622 3127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:27.988492 3127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5096a9180bf944f2b09b5daa98ba7521: Sending DeleteTablet for 3 replicas of tablet 4f89de3244f34bec929113684854c5c4
I20260806 03:42:27.988865 3255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89de3244f34bec929113684854c5c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:27 UTC) from {username='slave'} at 127.0.0.1:40910
I20260806 03:42:27.988940 3527 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89de3244f34bec929113684854c5c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:27 UTC) from {username='slave'} at 127.0.0.1:52496
I20260806 03:42:27.989106 3632 tablet_replica.cc:330] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: stopping tablet replica
I20260806 03:42:27.989106 3393 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89de3244f34bec929113684854c5c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:27 UTC) from {username='slave'} at 127.0.0.1:58910
I20260806 03:42:27.989235 3632 raft_consensus.cc:2229] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:27.989344 3632 raft_consensus.cc:2258] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:27.989358 3634 tablet_replica.cc:330] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: stopping tablet replica
I20260806 03:42:27.989457 3634 raft_consensus.cc:2229] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:27.989571 3634 raft_consensus.cc:2258] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:27.989749 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3190
I20260806 03:42:27.989799 3632 ts_tablet_manager.cc:1905] T 4f89de3244f34bec929113684854c5c4 P 95c79c9638f14eb7b46530b5eb77ddf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:27.991163 3634 ts_tablet_manager.cc:1905] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:27.992734 3634 ts_tablet_manager.cc:1918] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:27.992805 3634 log.cc:1196] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/4f89de3244f34bec929113684854c5c4
I20260806 03:42:27.993045 3634 ts_tablet_manager.cc:1939] T 4f89de3244f34bec929113684854c5c4 P 0988695b157b46ffbd950a438210758e: Deleting consensus metadata
I20260806 03:42:27.993512 3114 catalog_manager.cc:4841] TS 0988695b157b46ffbd950a438210758e (127.30.24.2:45519): tablet 4f89de3244f34bec929113684854c5c4 (table test-table [id=e06e1cb951594402952502323cba788e]) successfully deleted
I20260806 03:42:27.994776 3633 tablet_replica.cc:330] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: stopping tablet replica
I20260806 03:42:27.994908 3633 raft_consensus.cc:2229] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:27.995029 3633 raft_consensus.cc:2258] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:27.995484 3633 ts_tablet_manager.cc:1905] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:27.997066 3633 ts_tablet_manager.cc:1918] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:27.997182 3633 log.cc:1196] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/4f89de3244f34bec929113684854c5c4
I20260806 03:42:27.997462 3633 ts_tablet_manager.cc:1939] T 4f89de3244f34bec929113684854c5c4 P 6cd6a9f8df344972bb85fccfab2ef578: Deleting consensus metadata
I20260806 03:42:27.997892 3115 catalog_manager.cc:4841] TS 6cd6a9f8df344972bb85fccfab2ef578 (127.30.24.3:44491): tablet 4f89de3244f34bec929113684854c5c4 (table test-table [id=e06e1cb951594402952502323cba788e]) successfully deleted
W20260806 03:42:27.998935 3114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:44923 (error 108)
I20260806 03:42:27.999204 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3327
I20260806 03:42:27.999486 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:27.999348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44923 (local address 127.0.0.1:40922)
0806 03:42:27.999393 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:27.999447 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 03:42:28.000751 3114 catalog_manager.cc:4584] TS 95c79c9638f14eb7b46530b5eb77ddf3 (127.30.24.1:44923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f89de3244f34bec929113684854c5c4: Network error: Client connection negotiation failed: client connection to 127.30.24.1:44923: connect: Connection refused (error 111)
I20260806 03:42:28.005419 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3463
I20260806 03:42:28.011606 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3097
2026-08-06T03:42:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (663 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 03:42:28.032279 30816 mini_oidc.cc:152] Starting JWKS server
I20260806 03:42:28.033622 30816 webserver.cc:460] Webserver started at https://127.0.0.1:34495/ using document root <none> and password file <none>
I20260806 03:42:28.033706 30816 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 03:42:28.033833 30816 webserver.cc:460] Webserver started at http://127.0.0.1:35247/ using document root <none> and password file <none>
2026-08-06T03:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:28Z Disabled control of system clock
I20260806 03:42:28.043197 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33001
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:33001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/testchainca.cert with env {}
W20260806 03:42:28.126447 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:28.126675 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:28.126698 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:28.127847 3641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:28.127888 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:28.127902 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:28.127914 3641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:28.127926 3641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 03:42:28.127938 3641 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 03:42:28.127950 3641 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34495/jwks.json
I20260806 03:42:28.129611 3641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:33001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:28.129834 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:28.130010 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:28.134922 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:28.134903 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:28.134958 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:28.135243 3641 server_base.cc:1034] running on GCE node
I20260806 03:42:28.135524 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:28.135828 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:28.136970 3641 hybrid_clock.cc:648] HybridClock initialized: now 1785987748136941 us; error 40 us; skew 500 ppm
I20260806 03:42:28.138207 3641 webserver.cc:460] Webserver started at http://127.30.24.62:38833/ using document root <none> and password file <none>
I20260806 03:42:28.138463 3641 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:28.138535 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:28.138701 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:28.139819 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "d73f8cd228e042688ca668032444cc61"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "dad12bffadaa5fcea167a3c38be2fcc9"
server_key_iv: "832d477d8e8c0ecf8bd42b2f75f89cd7"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.140204 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "d73f8cd228e042688ca668032444cc61"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "dad12bffadaa5fcea167a3c38be2fcc9"
server_key_iv: "832d477d8e8c0ecf8bd42b2f75f89cd7"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.141606 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:28.142409 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:28.142592 3641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:28.142714 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "d73f8cd228e042688ca668032444cc61"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "dad12bffadaa5fcea167a3c38be2fcc9"
server_key_iv: "832d477d8e8c0ecf8bd42b2f75f89cd7"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.142804 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:28.162695 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:28.163113 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:28.163300 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:28.167781 3641 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:33001
I20260806 03:42:28.167817 3711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:33001 every 8 connection(s)
I20260806 03:42:28.168216 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:28.169333 3712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:28.171742 3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61: Bootstrap starting.
I20260806 03:42:28.172600 3712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.172974 3712 log.cc:824] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:28.173727 3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61: No bootstrap required, opened a new log
I20260806 03:42:28.175434 3712 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73f8cd228e042688ca668032444cc61" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33001 } }
I20260806 03:42:28.175614 3712 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.175662 3712 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73f8cd228e042688ca668032444cc61, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.175818 3712 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73f8cd228e042688ca668032444cc61" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33001 } }
I20260806 03:42:28.175904 3712 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:28.175944 3712 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:28.175999 3712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.176635 3712 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73f8cd228e042688ca668032444cc61" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33001 } }
I20260806 03:42:28.176760 3712 leader_election.cc:302] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73f8cd228e042688ca668032444cc61; no voters:
I20260806 03:42:28.176973 3712 leader_election.cc:288] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:28.177027 3715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.177249 3712 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:28.177256 3715 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [term 1 LEADER]: Becoming Leader. State: Replica: d73f8cd228e042688ca668032444cc61, State: Running, Role: LEADER
I20260806 03:42:28.177389 3715 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73f8cd228e042688ca668032444cc61" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33001 } }
I20260806 03:42:28.177806 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73f8cd228e042688ca668032444cc61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73f8cd228e042688ca668032444cc61" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33001 } } }
I20260806 03:42:28.177910 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.177911 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3641
I20260806 03:42:28.177894 3717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73f8cd228e042688ca668032444cc61. Latest consensus state: current_term: 1 leader_uuid: "d73f8cd228e042688ca668032444cc61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73f8cd228e042688ca668032444cc61" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33001 } } }
I20260806 03:42:28.177968 3717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.178009 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:28.178164 3719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:28.178216 30816 external_mini_cluster.cc:1428] Setting key f0fb01d5878075e48b4d89e9a1c8d6e3
I20260806 03:42:28.178725 3719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:28.181165 3719 catalog_manager.cc:1347] Generated new cluster ID: b5902d6e5ee04975891959a6586f3e50
I20260806 03:42:28.181229 3719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:28.186131 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.178719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.30.24.62:33001)
0806 03:42:28.179847 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.179856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.180475 (+ 619us) server_negotiation.cc:407] Connection header received
0806 03:42:28.180646 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.180653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.180718 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.180869 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:28.181129 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.182546 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.183756 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.184392 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.184531 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.185362 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:28.185366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:28.185367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:28.185589 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:28.185632 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:28.185648 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:28.185710 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:28.185838 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 03:42:28.185922 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":78,"threads_started":1}
I20260806 03:42:28.187126 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:42:28.203255 3719 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:28.203860 3719 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:28.215013 3719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73f8cd228e042688ca668032444cc61: Generated new TSK 0
I20260806 03:42:28.215255 3719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:28.247027 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.223573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.30.24.62:33001)
0806 03:42:28.223654 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.223656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.223730 (+ 74us) server_negotiation.cc:407] Connection header received
0806 03:42:28.223838 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.223839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.223848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.223907 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0806 03:42:28.224075 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.224587 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.225707 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.225898 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.225948 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.226172 (+ 224us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 03:42:28.246765 (+ 20593us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 03:42:28.246945 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 03:42:28.247099 3724 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
E20260806 03:42:28.247483 3742 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 03:42:28.247574 3742 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260806 03:42:28.247608 3741 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:28.223446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33001 (local address 127.0.0.1:58044)
0806 03:42:28.223655 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:28.223667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:28.223742 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:28.223925 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:28.223926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:28.223928 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 03:42:28.224008 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.224010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.224688 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.224689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.225628 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.225630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.225969 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.225970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.226038 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.226061 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 03:42:28.246934 (+ 20873us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 03:42:28.247114 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.24.62:33001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.24.62:33001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 03:42:28.248361 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3641
2026-08-06T03:42:28Z chronyd exiting
I20260806 03:42:28.286526 30816 test_util.cc:175] -----------------------------------------------
I20260806 03:42:28.286623 30816 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785987724420712-30816-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (255 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 03:42:28.287544 30816 mini_oidc.cc:152] Starting JWKS server
I20260806 03:42:28.288270 30816 webserver.cc:460] Webserver started at https://127.0.0.1:35505/ using document root <none> and password file <none>
I20260806 03:42:28.288313 30816 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 03:42:28.288556 30816 webserver.cc:460] Webserver started at http://127.0.0.1:45703/ using document root <none> and password file <none>
2026-08-06T03:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:28Z Disabled control of system clock
I20260806 03:42:28.297452 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:46075
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:46075
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/testchainca.cert with env {}
W20260806 03:42:28.379451 3748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:28.379674 3748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:28.379698 3748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:28.380906 3748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:28.380954 3748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:28.380967 3748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:28.380980 3748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:28.380991 3748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 03:42:28.381004 3748 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 03:42:28.381016 3748 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35505/jwks.json
I20260806 03:42:28.382519 3748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:46075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:28.382725 3748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:28.382884 3748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:28.387696 3754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:28.387706 3753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:28.387706 3756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:28.388197 3748 server_base.cc:1034] running on GCE node
I20260806 03:42:28.388499 3748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:28.388821 3748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:28.390014 3748 hybrid_clock.cc:648] HybridClock initialized: now 1785987748390008 us; error 41 us; skew 500 ppm
I20260806 03:42:28.391459 3748 webserver.cc:460] Webserver started at http://127.30.24.62:34303/ using document root <none> and password file <none>
I20260806 03:42:28.391719 3748 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:28.391788 3748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:28.391937 3748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:28.393100 3748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "713e77fa24f1406d867757086aa5598e"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "787ae36899a2c1a9d964152d1fb19e89"
server_key_iv: "9d8fa27122163448d5e4f3dc350f0d97"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.393478 3748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "713e77fa24f1406d867757086aa5598e"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "787ae36899a2c1a9d964152d1fb19e89"
server_key_iv: "9d8fa27122163448d5e4f3dc350f0d97"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.394876 3748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:28.395637 3762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:28.395825 3748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:28.395949 3748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "713e77fa24f1406d867757086aa5598e"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "787ae36899a2c1a9d964152d1fb19e89"
server_key_iv: "9d8fa27122163448d5e4f3dc350f0d97"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.396036 3748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:28.414778 3748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:28.415091 3748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:28.415205 3748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:28.419303 3748 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:46075
I20260806 03:42:28.419373 3816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:46075 every 8 connection(s)
I20260806 03:42:28.419731 3748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:28.420414 3817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:28.421963 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3748
I20260806 03:42:28.422057 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:28.422253 30816 external_mini_cluster.cc:1428] Setting key 5250c942b388eb83f34e3f07359bb4a3
I20260806 03:42:28.423736 3817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e: Bootstrap starting.
I20260806 03:42:28.424503 3817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.424830 3817 log.cc:824] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:28.425619 3817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e: No bootstrap required, opened a new log
I20260806 03:42:28.427927 3817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713e77fa24f1406d867757086aa5598e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46075 } }
I20260806 03:42:28.428117 3817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.428156 3817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713e77fa24f1406d867757086aa5598e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.428278 3817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713e77fa24f1406d867757086aa5598e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46075 } }
I20260806 03:42:28.428360 3817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:28.428403 3817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:28.428480 3817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.428530 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.422800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.30.24.62:46075)
0806 03:42:28.423036 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.423045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.423079 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:42:28.423603 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.423612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.423691 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.423851 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:28.424091 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.425460 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.426291 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.426901 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.427032 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.427709 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:28.427713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:28.427713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:28.427924 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:28.427967 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:28.427981 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:28.428054 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:28.428179 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 03:42:28.428260 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260806 03:42:28.429190 3817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713e77fa24f1406d867757086aa5598e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46075 } }
I20260806 03:42:28.429286 3817 leader_election.cc:302] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 713e77fa24f1406d867757086aa5598e; no voters:
I20260806 03:42:28.429461 3817 leader_election.cc:288] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:28.429495 3822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.429626 3822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [term 1 LEADER]: Becoming Leader. State: Replica: 713e77fa24f1406d867757086aa5598e, State: Running, Role: LEADER
I20260806 03:42:28.429682 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:42:28.429719 3822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713e77fa24f1406d867757086aa5598e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46075 } }
I20260806 03:42:28.429749 3817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:28.430039 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "713e77fa24f1406d867757086aa5598e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713e77fa24f1406d867757086aa5598e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46075 } } }
I20260806 03:42:28.430107 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.430276 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713e77fa24f1406d867757086aa5598e. Latest consensus state: current_term: 1 leader_uuid: "713e77fa24f1406d867757086aa5598e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713e77fa24f1406d867757086aa5598e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46075 } } }
I20260806 03:42:28.430330 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.430766 3827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:28.431272 3827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:28.432868 3827 catalog_manager.cc:1347] Generated new cluster ID: 221cc1619d6340dc858a36a2cd12efc5
I20260806 03:42:28.432936 3827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:28.440762 3827 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:28.443854 3827 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:28.456130 3827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 713e77fa24f1406d867757086aa5598e: Generated new TSK 0
I20260806 03:42:28.456363 3827 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 03:42:28.492745 3847 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 03:42:28.492828 3847 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 03:42:28.493283 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.470836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.30.24.62:46075)
0806 03:42:28.470893 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.470896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.472678 (+ 1782us) server_negotiation.cc:407] Connection header received
0806 03:42:28.472776 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.472777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.472803 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.472876 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0806 03:42:28.473040 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.473873 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.475036 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.475260 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.475338 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.475523 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 03:42:28.493016 (+ 17493us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 03:42:28.493198 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260806 03:42:28.493337 3820 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 03:42:28.493372 3846 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:28.470747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46075 (local address 127.0.0.1:52002)
0806 03:42:28.472592 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:28.472621 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:28.472698 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:28.472888 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:28.472889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:28.472891 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 03:42:28.472956 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.472958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.473972 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.473975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.474947 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.474951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.475350 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.475351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.475418 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.475425 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 03:42:28.493155 (+ 17730us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 03:42:28.493264 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.24.62:46075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1812,"thread_start_us":80,"threads_started":1}
I20260806 03:42:28.493971 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3748
2026-08-06T03:42:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 03:42:28.533639 30816 mini_oidc.cc:152] Starting JWKS server
I20260806 03:42:28.534327 30816 webserver.cc:460] Webserver started at https://127.0.0.1:38405/ using document root <none> and password file <none>
I20260806 03:42:28.534368 30816 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 03:42:28.534482 30816 webserver.cc:460] Webserver started at http://127.0.0.1:41829/ using document root <none> and password file <none>
2026-08-06T03:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:28Z Disabled control of system clock
I20260806 03:42:28.543170 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38557
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 03:42:28.625603 3854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:28.625849 3854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:28.625887 3854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:28.627044 3854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:28.627113 3854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:28.627132 3854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:28.627153 3854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:28.627184 3854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 03:42:28.627209 3854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 03:42:28.627229 3854 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38405/jwks.json
I20260806 03:42:28.628604 3854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:28.628877 3854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:28.629168 3854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:28.634071 3862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:28.634068 3860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:28.634260 3854 server_base.cc:1034] running on GCE node
W20260806 03:42:28.634392 3859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:28.634668 3854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:28.634918 3854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:28.636091 3854 hybrid_clock.cc:648] HybridClock initialized: now 1785987748636091 us; error 43 us; skew 500 ppm
I20260806 03:42:28.637429 3854 webserver.cc:460] Webserver started at http://127.30.24.62:43587/ using document root <none> and password file <none>
I20260806 03:42:28.637682 3854 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:28.637756 3854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:28.637895 3854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:28.638976 3854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "127a39da15cf4330a3b306634626fc99"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "000ece8e6b3f07980c6851d18240e428"
server_key_iv: "dd83549a8cc4477fe9c608f42e7bd7d8"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.639374 3854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "127a39da15cf4330a3b306634626fc99"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "000ece8e6b3f07980c6851d18240e428"
server_key_iv: "dd83549a8cc4477fe9c608f42e7bd7d8"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.640775 3854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:28.641606 3868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:28.641801 3854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:28.641911 3854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "127a39da15cf4330a3b306634626fc99"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "000ece8e6b3f07980c6851d18240e428"
server_key_iv: "dd83549a8cc4477fe9c608f42e7bd7d8"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.641987 3854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:28.654323 3854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:28.654635 3854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:28.654784 3854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:28.659020 3854 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:38557
I20260806 03:42:28.659055 3920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:38557 every 8 connection(s)
I20260806 03:42:28.659397 3854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:28.660061 3921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:28.662760 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99: Bootstrap starting.
I20260806 03:42:28.663309 3921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.663545 3921 log.cc:824] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:28.664215 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99: No bootstrap required, opened a new log
I20260806 03:42:28.665853 3921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127a39da15cf4330a3b306634626fc99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38557 } }
I20260806 03:42:28.665993 3921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.666015 3921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127a39da15cf4330a3b306634626fc99, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.666162 3921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127a39da15cf4330a3b306634626fc99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38557 } }
I20260806 03:42:28.666244 3921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:28.666267 3921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:28.666296 3921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.666785 3921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127a39da15cf4330a3b306634626fc99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38557 } }
I20260806 03:42:28.666870 3921 leader_election.cc:302] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127a39da15cf4330a3b306634626fc99; no voters:
I20260806 03:42:28.666986 3921 leader_election.cc:288] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:28.667114 3924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.667265 3924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [term 1 LEADER]: Becoming Leader. State: Replica: 127a39da15cf4330a3b306634626fc99, State: Running, Role: LEADER
I20260806 03:42:28.667245 3921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:28.667372 3924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127a39da15cf4330a3b306634626fc99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38557 } }
I20260806 03:42:28.667677 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3854
I20260806 03:42:28.667802 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:28.667749 3925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127a39da15cf4330a3b306634626fc99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127a39da15cf4330a3b306634626fc99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38557 } } }
I20260806 03:42:28.667848 3925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.667933 3926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127a39da15cf4330a3b306634626fc99. Latest consensus state: current_term: 1 leader_uuid: "127a39da15cf4330a3b306634626fc99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127a39da15cf4330a3b306634626fc99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38557 } } }
I20260806 03:42:28.667991 3926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.668010 30816 external_mini_cluster.cc:1428] Setting key 2a24e4a441152db226427bfba86ace02
I20260806 03:42:28.668155 3931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:28.668705 3931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:28.670317 3931 catalog_manager.cc:1347] Generated new cluster ID: 2fd52cedd01340ee97febf5e52f0e447
I20260806 03:42:28.670385 3931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:28.673944 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.668492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.30.24.62:38557)
0806 03:42:28.668743 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.668806 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.669343 (+ 537us) server_negotiation.cc:407] Connection header received
0806 03:42:28.669470 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.669478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.669544 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.669705 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:28.669962 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.671087 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.671896 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.672338 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.672496 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.673255 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:28.673259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:28.673259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:28.673461 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:28.673494 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:28.673505 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:28.673588 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:28.673657 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:42:28.673738 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260806 03:42:28.674875 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:42:28.679998 3931 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:28.680572 3931 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:28.685813 3931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127a39da15cf4330a3b306634626fc99: Generated new TSK 0
I20260806 03:42:28.685992 3931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:28.698138 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.694431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.30.24.62:38557)
0806 03:42:28.694479 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.694481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.694577 (+ 96us) server_negotiation.cc:407] Connection header received
0806 03:42:28.694618 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.694619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.694633 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.694716 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0806 03:42:28.694845 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.695655 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.696375 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.696723 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.696798 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.696947 (+ 149us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 03:42:28.697919 (+ 972us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 03:42:28.698057 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260806 03:42:28.698201 3940 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 03:42:28.700055 3950 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:28.694346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38557 (local address 127.0.0.1:38828)
0806 03:42:28.694496 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:28.694506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:28.694569 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:28.694728 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:28.694729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:28.694731 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 03:42:28.694778 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.694779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.695737 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.695738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.696295 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.696297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.696803 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.696804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.696847 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.696851 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 03:42:28.698023 (+ 1172us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 03:42:28.699959 (+ 1936us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.24.62:38557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1736,"thread_start_us":101,"threads_started":1}
I20260806 03:42:28.700413 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3854
E20260806 03:42:28.709447 3951 webserver.cc:533] Webserver: sslize failed:
E20260806 03:42:28.709506 3951 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T03:42:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T03:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:28Z Disabled control of system clock
I20260806 03:42:28.748433 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:45753
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:45753
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/testchainca.cert with env {}
W20260806 03:42:28.831317 3955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:28.831534 3955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:28.831564 3955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:28.832823 3955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:28.832864 3955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:28.832877 3955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:28.832890 3955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:28.832944 3955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 03:42:28.832964 3955 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 03:42:28.832989 3955 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 03:42:28.834514 3955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:45753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:28.834789 3955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:28.835024 3955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:28.839983 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:28.840128 3955 server_base.cc:1034] running on GCE node
W20260806 03:42:28.840137 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:28.839987 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:28.840600 3955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:28.840902 3955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:28.842072 3955 hybrid_clock.cc:648] HybridClock initialized: now 1785987748842062 us; error 40 us; skew 500 ppm
I20260806 03:42:28.843438 3955 webserver.cc:460] Webserver started at http://127.30.24.62:40441/ using document root <none> and password file <none>
I20260806 03:42:28.843708 3955 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:28.843786 3955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:28.843937 3955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:28.845091 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "626dc4e852ac435699a5bebb99b8ffa6"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "51c3bf1c19f37593a83ca50f0400db97"
server_key_iv: "065740de1051338b91f8b9ef095d0cb3"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.845479 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "626dc4e852ac435699a5bebb99b8ffa6"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "51c3bf1c19f37593a83ca50f0400db97"
server_key_iv: "065740de1051338b91f8b9ef095d0cb3"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.846813 3955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 03:42:28.847591 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:28.847790 3955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:28.847905 3955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "626dc4e852ac435699a5bebb99b8ffa6"
format_stamp: "Formatted at 2026-08-06 03:42:28 on dist-test-slave-10pc"
server_key: "51c3bf1c19f37593a83ca50f0400db97"
server_key_iv: "065740de1051338b91f8b9ef095d0cb3"
server_key_version: "encryptionkey@0"
I20260806 03:42:28.847999 3955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:28.862300 3955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:28.862612 3955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:28.862766 3955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:28.866776 3955 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:45753
I20260806 03:42:28.866809 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:45753 every 8 connection(s)
I20260806 03:42:28.867166 3955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:28.868209 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:28.870612 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6: Bootstrap starting.
I20260806 03:42:28.871254 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.871555 4026 log.cc:824] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:28.872365 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6: No bootstrap required, opened a new log
I20260806 03:42:28.873348 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 3955
I20260806 03:42:28.873436 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:28.873652 30816 external_mini_cluster.cc:1428] Setting key 7be9953633d95fb982168f252e2af1bd
I20260806 03:42:28.874441 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626dc4e852ac435699a5bebb99b8ffa6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45753 } }
I20260806 03:42:28.874631 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.874677 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626dc4e852ac435699a5bebb99b8ffa6, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.874809 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626dc4e852ac435699a5bebb99b8ffa6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45753 } }
I20260806 03:42:28.874905 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:28.874951 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:28.874998 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.875582 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626dc4e852ac435699a5bebb99b8ffa6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45753 } }
I20260806 03:42:28.875713 4026 leader_election.cc:302] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626dc4e852ac435699a5bebb99b8ffa6; no voters:
I20260806 03:42:28.875912 4026 leader_election.cc:288] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:28.875936 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.876053 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [term 1 LEADER]: Becoming Leader. State: Replica: 626dc4e852ac435699a5bebb99b8ffa6, State: Running, Role: LEADER
I20260806 03:42:28.876140 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626dc4e852ac435699a5bebb99b8ffa6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45753 } }
I20260806 03:42:28.876204 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:28.876487 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626dc4e852ac435699a5bebb99b8ffa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626dc4e852ac435699a5bebb99b8ffa6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45753 } } }
I20260806 03:42:28.876502 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626dc4e852ac435699a5bebb99b8ffa6. Latest consensus state: current_term: 1 leader_uuid: "626dc4e852ac435699a5bebb99b8ffa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626dc4e852ac435699a5bebb99b8ffa6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45753 } } }
I20260806 03:42:28.876634 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.876653 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:28.877523 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:28.878109 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:28.879699 4039 catalog_manager.cc:1347] Generated new cluster ID: ac572e22fb8e4f6bb558e0f528167013
I20260806 03:42:28.879755 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:28.880211 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.874236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.30.24.62:45753)
0806 03:42:28.874477 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.874486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.874519 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:42:28.874642 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.874650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.874748 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.874894 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:28.875144 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.876771 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.877966 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.878486 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.878626 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.879507 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:28.879511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:28.879511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:28.879739 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:28.879791 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:28.879808 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:28.879872 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:28.879939 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 03:42:28.880017 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260806 03:42:28.884538 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:42:28.890466 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:28.890997 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:28.893615 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626dc4e852ac435699a5bebb99b8ffa6: Generated new TSK 0
I20260806 03:42:28.893792 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:29.369858 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:28.903259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.30.24.62:45753)
0806 03:42:28.903326 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:28.903328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:28.903467 (+ 139us) server_negotiation.cc:407] Connection header received
0806 03:42:28.903543 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:28.903544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:28.903555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:28.903615 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0806 03:42:28.903769 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.904266 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.905415 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.905615 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.905669 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.905869 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 03:42:29.369596 (+463727us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 03:42:29.369794 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 03:42:29.369961 4030 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 03:42:29.370057 4055 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:28.903211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:45753 (local address 127.0.0.1:46816)
0806 03:42:28.903387 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:28.903400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:28.903473 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:28.903636 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:28.903637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:28.903639 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 03:42:28.903702 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.903704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.904347 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.904348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.905333 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:28.905337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:28.905699 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:28.905700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:28.905764 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:28.905769 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 03:42:29.369781 (+464012us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 03:42:29.369919 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.24.62:45753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260806 03:42:29.370461 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 3955
2026-08-06T03:42:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (650 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (104 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (92 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:29 dist-test-slave-10pc krb5kdc[4067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:29 dist-test-slave-10pc krb5kdc[4067](info): set up 2 sockets
Aug 06 03:42:29 dist-test-slave-10pc krb5kdc[4067](info): commencing operation
krb5kdc: starting...
W20260806 03:42:30.662006 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:30 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:30.791365 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:45075
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:30.873980 4081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:30.874191 4081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:30.874214 4081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:30.875382 4081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:30.875425 4081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:30.875440 4081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:30.875452 4081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:30.875465 4081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:30.877029 4081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:30.877300 4081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:30.877538 4081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:30.882567 4087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:30.882561 4086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:30.882556 4089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:30.882705 4081 server_base.cc:1034] running on GCE node
I20260806 03:42:30.883097 4081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:30.883358 4081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:30.884521 4081 hybrid_clock.cc:648] HybridClock initialized: now 1785987750884504 us; error 39 us; skew 500 ppm
Aug 06 03:42:30 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:30.887179 4081 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:30.887650 4081 webserver.cc:460] Webserver started at http://127.30.24.62:38411/ using document root <none> and password file <none>
I20260806 03:42:30.887934 4081 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:30.888017 4081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:30.888181 4081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:30.889401 4081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "43f695d9dece4d4f99ce893361b2ebb6"
format_stamp: "Formatted at 2026-08-06 03:42:30 on dist-test-slave-10pc"
server_key: "6ca0e726b93e2479f453d2b00e622d89"
server_key_iv: "1d2827e89819833ed7279b4b57a5aa33"
server_key_version: "encryptionkey@0"
I20260806 03:42:30.889765 4081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "43f695d9dece4d4f99ce893361b2ebb6"
format_stamp: "Formatted at 2026-08-06 03:42:30 on dist-test-slave-10pc"
server_key: "6ca0e726b93e2479f453d2b00e622d89"
server_key_iv: "1d2827e89819833ed7279b4b57a5aa33"
server_key_version: "encryptionkey@0"
I20260806 03:42:30.891037 4081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:42:30.891789 4097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:30.891973 4081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:30.892081 4081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "43f695d9dece4d4f99ce893361b2ebb6"
format_stamp: "Formatted at 2026-08-06 03:42:30 on dist-test-slave-10pc"
server_key: "6ca0e726b93e2479f453d2b00e622d89"
server_key_iv: "1d2827e89819833ed7279b4b57a5aa33"
server_key_version: "encryptionkey@0"
I20260806 03:42:30.892159 4081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:30.905630 4081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:30.907545 4081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:30.907724 4081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:30.911825 4081 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:45075
I20260806 03:42:30.911861 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:45075 every 8 connection(s)
I20260806 03:42:30.912215 4081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:30.913072 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:30.915382 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6: Bootstrap starting.
I20260806 03:42:30.915977 4150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.916291 4150 log.cc:824] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:30.916702 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4081
I20260806 03:42:30.916817 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:30.917016 30816 external_mini_cluster.cc:1428] Setting key 468acd0c93140e53de79f89a244807a3
I20260806 03:42:30.917197 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6: No bootstrap required, opened a new log
I20260806 03:42:30.919615 4150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f695d9dece4d4f99ce893361b2ebb6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45075 } }
I20260806 03:42:30.919780 4150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.919826 4150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f695d9dece4d4f99ce893361b2ebb6, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.919962 4150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f695d9dece4d4f99ce893361b2ebb6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45075 } }
I20260806 03:42:30.920043 4150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:30.920087 4150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:30.920142 4150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:30.920763 4150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f695d9dece4d4f99ce893361b2ebb6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45075 } }
I20260806 03:42:30.920919 4150 leader_election.cc:302] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43f695d9dece4d4f99ce893361b2ebb6; no voters:
I20260806 03:42:30.921098 4150 leader_election.cc:288] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:30.921129 4155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:30.921247 4155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [term 1 LEADER]: Becoming Leader. State: Replica: 43f695d9dece4d4f99ce893361b2ebb6, State: Running, Role: LEADER
I20260806 03:42:30.921388 4155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f695d9dece4d4f99ce893361b2ebb6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45075 } }
I20260806 03:42:30.921391 4150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:30.922484 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43f695d9dece4d4f99ce893361b2ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f695d9dece4d4f99ce893361b2ebb6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45075 } } }
I20260806 03:42:30.922587 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:30.923521 4157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f695d9dece4d4f99ce893361b2ebb6. Latest consensus state: current_term: 1 leader_uuid: "43f695d9dece4d4f99ce893361b2ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f695d9dece4d4f99ce893361b2ebb6" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 45075 } } }
I20260806 03:42:30.923677 4157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6 [sys.catalog]: This master's current role is: LEADER
W20260806 03:42:30.923418 4172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:42:30.923781 4172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:42:30.923856 4166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:30.924108 4166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 03:42:30 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:30.925973 4166 catalog_manager.cc:1347] Generated new cluster ID: e5aa482377aa48788e7ab1c699bd7467
I20260806 03:42:30.926025 4166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:30.929502 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:30.917571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.30.24.62:45075)
0806 03:42:30.917771 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:30.917780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:30.917815 (+ 35us) server_negotiation.cc:407] Connection header received
0806 03:42:30.918525 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:30.918533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:30.918581 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:30.918695 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:30.919517 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:30.920946 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:30.921961 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:30.922641 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:30.922771 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:30.925665 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:30.925669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:30.925670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:30.925683 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:30.927802 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:30.928210 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:30.928213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:30.928213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:30.928332 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:30.928548 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:30.928549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:30.928549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:30.928876 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:30.929046 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:30.929137 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 03:42:30.929281 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260806 03:42:30.934259 4166 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:30.934782 4166 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:30.938614 4166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43f695d9dece4d4f99ce893361b2ebb6: Generated new TSK 0
I20260806 03:42:30.938865 4166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:30.967023 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:45075
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:31.052811 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:31.053038 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:31.053063 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:31.054431 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:31.054492 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:31.054522 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:31.056087 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:31.056350 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:31.056635 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:31.062072 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:31.062075 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:31.062219 4177 server_base.cc:1034] running on GCE node
W20260806 03:42:31.062075 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:31.062620 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:31.062875 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:31.064045 4177 hybrid_clock.cc:648] HybridClock initialized: now 1785987751064015 us; error 64 us; skew 500 ppm
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.066758 4177 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:31.067272 4177 webserver.cc:460] Webserver started at http://127.30.24.1:39837/ using document root <none> and password file <none>
I20260806 03:42:31.067518 4177 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:31.067595 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:31.067754 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:31.068940 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "c9ab1c9df16d42db934df1f91e7d4d27"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "4bf06743c35f34486223d2cdacd81874"
server_key_iv: "eddf09fc7a9e01358273af702bb759c0"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.069324 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "c9ab1c9df16d42db934df1f91e7d4d27"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "4bf06743c35f34486223d2cdacd81874"
server_key_iv: "eddf09fc7a9e01358273af702bb759c0"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.070703 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:31.071478 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.071656 4177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:31.071768 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "c9ab1c9df16d42db934df1f91e7d4d27"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "4bf06743c35f34486223d2cdacd81874"
server_key_iv: "eddf09fc7a9e01358273af702bb759c0"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.071846 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:31.095369 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:31.098039 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:31.098227 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:31.098505 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:31.098852 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:31.098909 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.098946 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:31.098994 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.105057 4177 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:40855
I20260806 03:42:31.105099 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:40855 every 8 connection(s)
I20260806 03:42:31.105433 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:31.112789 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4177
I20260806 03:42:31.112898 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:31.113108 30816 external_mini_cluster.cc:1428] Setting key 61da4d69e9751e624809f8e786f2325e
I20260806 03:42:31.115427 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.106191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:37115 (local address 127.30.24.62:45075)
0806 03:42:31.106237 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.106239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.106426 (+ 187us) server_negotiation.cc:407] Connection header received
0806 03:42:31.107396 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.107397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.107406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.107473 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.108142 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.108726 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.109731 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.109927 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.109984 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.112212 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:31.112214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:31.112214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:31.112221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:31.113959 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.114385 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.114386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.114387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.114454 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.114734 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.114735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.114735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.114858 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:31.114932 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:31.115238 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 03:42:31.115335 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:42:31.115626 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.106123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:45075 (local address 127.30.24.1:37115)
0806 03:42:31.106297 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:31.106341 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:31.107304 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:31.107564 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:31.107567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:31.107819 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:31.108064 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:31.108069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.108808 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.108809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:31.109647 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:31.109650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.110037 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.110038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:31.110140 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.110613 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:31.110621 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:31.112121 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:31.114039 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:31.114046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:31.114049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:31.114288 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:31.114524 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:31.114524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:31.114525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:31.114641 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:31.114939 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:31.114947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:31.115098 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 03:42:31.115202 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 03:42:31.115366 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260806 03:42:31.116204 4307 heartbeater.cc:347] Connected to a master server at 127.30.24.62:45075
I20260806 03:42:31.116323 4307 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:31.116568 4307 heartbeater.cc:511] Master 127.30.24.62:45075 requested a full tablet report, sending...
I20260806 03:42:31.117121 4114 ts_manager.cc:194] Registered new tserver with Master: c9ab1c9df16d42db934df1f91e7d4d27 (127.30.24.1:40855)
I20260806 03:42:31.117739 4114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:37115
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:31.143067 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:45075
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:31.227248 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:31.227463 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:31.227487 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:31.228751 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:31.228792 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:31.228819 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:31.230329 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:31.230594 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:31.230834 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:31.235931 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:31.235953 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:31.236117 4312 server_base.cc:1034] running on GCE node
W20260806 03:42:31.235931 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:31.236421 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:31.236727 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:31.237920 4312 hybrid_clock.cc:648] HybridClock initialized: now 1785987751237897 us; error 58 us; skew 500 ppm
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.240762 4312 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:31.241242 4312 webserver.cc:460] Webserver started at http://127.30.24.2:40161/ using document root <none> and password file <none>
I20260806 03:42:31.241492 4312 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:31.241572 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:31.241722 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:31.242836 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "bca2862d562341498eac16743c5e527d"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "f99ae5e8b93aa36beec50bbfd67f2a87"
server_key_iv: "0a2cd2839086f5c5dbcf0537c13a1ba2"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.243214 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "bca2862d562341498eac16743c5e527d"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "f99ae5e8b93aa36beec50bbfd67f2a87"
server_key_iv: "0a2cd2839086f5c5dbcf0537c13a1ba2"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.244679 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:31.245559 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.245749 4312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:31.245874 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "bca2862d562341498eac16743c5e527d"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "f99ae5e8b93aa36beec50bbfd67f2a87"
server_key_iv: "0a2cd2839086f5c5dbcf0537c13a1ba2"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.245959 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:31.257860 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:31.260290 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:31.260504 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:31.260782 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:31.261108 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:31.261163 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.261198 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:31.261245 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.267138 4312 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:35389
I20260806 03:42:31.267195 4441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:35389 every 8 connection(s)
I20260806 03:42:31.267514 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:31.268939 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4312
I20260806 03:42:31.269016 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:31.269225 30816 external_mini_cluster.cc:1428] Setting key d3b0cfc293108941c4ef2195fc5500ad
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:31.278345 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.268287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:57497 (local address 127.30.24.62:45075)
0806 03:42:31.268379 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.268381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.268553 (+ 172us) server_negotiation.cc:407] Connection header received
0806 03:42:31.269720 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.269721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.269734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.269785 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.270501 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.271005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.272409 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.272682 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.272747 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.275278 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:31.275280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:31.275281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:31.275289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:31.276910 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.277338 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.277341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.277342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.277400 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.277671 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.277673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.277673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.277794 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:31.277868 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:31.278155 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 03:42:31.278246 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 03:42:31.278549 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.268255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:45075 (local address 127.30.24.2:57497)
0806 03:42:31.268407 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:31.268471 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:31.269608 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:31.269863 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:31.269866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:31.270208 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:31.270406 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:31.270411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.271110 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.271111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:31.272314 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:31.272317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.272776 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.272777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:31.272932 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.273473 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:31.273484 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:31.275176 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:31.276999 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:31.277007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:31.277009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:31.277247 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:31.277469 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:31.277470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:31.277470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:31.277598 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:31.277870 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:31.277879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:31.278023 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 03:42:31.278141 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 03:42:31.278307 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
I20260806 03:42:31.279101 4442 heartbeater.cc:347] Connected to a master server at 127.30.24.62:45075
I20260806 03:42:31.279217 4442 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:31.279399 4442 heartbeater.cc:511] Master 127.30.24.62:45075 requested a full tablet report, sending...
I20260806 03:42:31.279891 4114 ts_manager.cc:194] Registered new tserver with Master: bca2862d562341498eac16743c5e527d (127.30.24.2:35389)
I20260806 03:42:31.280285 4114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:57497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:31.299592 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:45075
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:31.384116 4447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:31.384366 4447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:31.384404 4447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:31.385627 4447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:31.385697 4447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:31.385743 4447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:31.387257 4447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:31.387531 4447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:31.387764 4447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:31.393002 4455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:31.393008 4453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:31.393054 4452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:31.393237 4447 server_base.cc:1034] running on GCE node
I20260806 03:42:31.393483 4447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:31.393738 4447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:31.394862 4447 hybrid_clock.cc:648] HybridClock initialized: now 1785987751394825 us; error 59 us; skew 500 ppm
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.397737 4447 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:31.398144 4447 webserver.cc:460] Webserver started at http://127.30.24.3:40341/ using document root <none> and password file <none>
I20260806 03:42:31.398335 4447 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:31.398375 4447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:31.398463 4447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:31.399475 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "055570b730b14d789e4384a2e87d7ba3"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "baeada5ab6e5e74f30353cba64a29c65"
server_key_iv: "a146694f73d9d1bdf4ac2a659462b6d5"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.399789 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "055570b730b14d789e4384a2e87d7ba3"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "baeada5ab6e5e74f30353cba64a29c65"
server_key_iv: "a146694f73d9d1bdf4ac2a659462b6d5"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.401373 4447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:31.402125 4462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.402303 4447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:31.402413 4447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "055570b730b14d789e4384a2e87d7ba3"
format_stamp: "Formatted at 2026-08-06 03:42:31 on dist-test-slave-10pc"
server_key: "baeada5ab6e5e74f30353cba64a29c65"
server_key_iv: "a146694f73d9d1bdf4ac2a659462b6d5"
server_key_version: "encryptionkey@0"
I20260806 03:42:31.402493 4447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:31.440127 4447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:31.442544 4447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:31.442728 4447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:31.442994 4447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:31.443337 4447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:31.443394 4447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.443440 4447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:31.443470 4447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:31.449714 4447 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:39371
I20260806 03:42:31.449745 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:39371 every 8 connection(s)
I20260806 03:42:31.450172 4447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:31.459005 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4447
I20260806 03:42:31.459095 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:31.459263 30816 external_mini_cluster.cc:1428] Setting key 90c0f0709ccfcd651a1f16904e88b64f
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:31.465543 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.450959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:50499 (local address 127.30.24.62:45075)
0806 03:42:31.451021 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.451023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.451274 (+ 251us) server_negotiation.cc:407] Connection header received
0806 03:42:31.452210 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.452211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.452223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.452281 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.453166 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.453929 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.455407 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.455687 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.455727 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.462404 (+ 6677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:31.462405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:31.462406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:31.462412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:31.464112 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.464561 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.464562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.464563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.464626 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.464864 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.464864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.464864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.464979 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:31.465080 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:31.465397 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 03:42:31.465476 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 03:42:31.465663 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.450932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:45075 (local address 127.30.24.3:50499)
0806 03:42:31.451099 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:31.451147 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:31.452110 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:31.452336 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:31.452340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:31.452706 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:31.453012 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:31.453018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.454053 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.454054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:31.455302 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:31.455305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.455817 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.455818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:31.455960 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.460756 (+ 4796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:31.460770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:31.462311 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:31.464190 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:31.464197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:31.464199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:31.464423 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:31.464687 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:31.464688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:31.464688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:31.464799 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:31.465075 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:31.465080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:31.465233 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 03:42:31.465316 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 03:42:31.465447 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260806 03:42:31.466121 4580 heartbeater.cc:347] Connected to a master server at 127.30.24.62:45075
I20260806 03:42:31.466212 4580 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:31.466364 4580 heartbeater.cc:511] Master 127.30.24.62:45075 requested a full tablet report, sending...
I20260806 03:42:31.466709 4114 ts_manager.cc:194] Registered new tserver with Master: 055570b730b14d789e4384a2e87d7ba3 (127.30.24.3:39371)
I20260806 03:42:31.467108 4114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:50499
I20260806 03:42:31.471241 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/test-admin.keytab.
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.489259 30816 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 03:42:31.491015 4587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:31.490216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.30.24.1:40855)
0806 03:42:31.490409 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.490415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.490425 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:31.490456 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.490458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.490472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.490540 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.490923 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33694: BlockingRecv error: recv got EOF from 127.0.0.1:33694 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
W20260806 03:42:31.491227 30816 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc' of type FILE
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.492218 30816 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.493112 30816 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:31.501159 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.493605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.30.24.1:40855)
0806 03:42:31.493662 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.493664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.493716 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:42:31.493764 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.493764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.493774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.493818 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.494264 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.495156 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.495938 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.496278 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.496335 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.498396 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:31.498404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:31.498404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:31.498410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:31.500040 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.500341 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.500344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.500344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.500398 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.500614 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.500615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.500615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.500738 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:31.500850 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:31.500906 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 03:42:31.500969 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 03:42:31.502985 4587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:31.502432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.30.24.1:40855)
0806 03:42:31.502506 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.502508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.502561 (+ 53us) server_negotiation.cc:407] Connection header received
0806 03:42:31.502622 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.502623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.502632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.502669 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.502940 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33708: BlockingRecv error: recv got EOF from 127.0.0.1:33708 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 03:42:31.503088 30816 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc)
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.504046 30816 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.505038 30816 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:31.512102 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.505617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.30.24.1:40855)
0806 03:42:31.505683 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.505685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.505697 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:31.505763 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.505764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.505775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.505818 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.506191 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.506686 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.507389 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.507615 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.507693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.509732 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:31.509733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:31.509734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:31.509741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:31.510978 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.511318 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.511319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.511320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.511380 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.511582 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.511582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.511583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.511693 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:31.511788 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:31.511925 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 03:42:31.512007 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 03:42:31.513782 4587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:31.513183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.30.24.1:40855)
0806 03:42:31.513250 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.513252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.513335 (+ 83us) server_negotiation.cc:407] Connection header received
0806 03:42:31.513407 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.513408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.513417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.513471 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.513735 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33720: BlockingRecv error: recv got EOF from 127.0.0.1:33720 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 03:42:31.513878 30816 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785987724420712-30816-0/krb5kdc/krb5cc' of type FILE
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.514834 30816 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:31.515645 30816 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:42:31.522490 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:31.516102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.30.24.1:40855)
0806 03:42:31.516183 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:31.516184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:31.516263 (+ 79us) server_negotiation.cc:407] Connection header received
0806 03:42:31.516288 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:31.516289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:31.516297 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:31.516358 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:31.516773 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.517275 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.517980 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:31.518194 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:31.518250 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:31.520134 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:31.520135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:31.520136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:31.520142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:31.521401 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.521790 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.521791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.521792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.521868 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:31.522036 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:31.522037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:31.522037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:31.522117 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:31.522200 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:31.522319 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 03:42:31.522385 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 03:42:31.523248 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4177
I20260806 03:42:31.529325 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4312
I20260806 03:42:31.534798 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4447
I20260806 03:42:31.540285 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4081
2026-08-06T03:42:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1972 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4600](info): set up 2 sockets
Aug 06 03:42:31 dist-test-slave-10pc krb5kdc[4600](info): commencing operation
krb5kdc: starting...
W20260806 03:42:32.589862 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:32 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab.
Entry for principal oryx/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab.
I20260806 03:42:32.713114 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42619
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab
--principal=oryx/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:42619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:32.794919 4613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:32.795172 4613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:32.795212 4613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:32.796355 4613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:32.796422 4613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:32.796445 4613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:32.796468 4613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:32.796517 4613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:32.798120 4613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:42619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab
--principal=oryx/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:32.798400 4613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:32.798640 4613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:32.803457 4618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:32.803563 4613 server_base.cc:1034] running on GCE node
W20260806 03:42:32.803444 4621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:32.803447 4619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:32.804021 4613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:32.804268 4613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:32.805411 4613 hybrid_clock.cc:648] HybridClock initialized: now 1785987752805381 us; error 46 us; skew 500 ppm
Aug 06 03:42:32 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:32.807920 4613 init.cc:377] Logged in from keytab as oryx/127.30.24.62@KRBTEST.COM (short username oryx)
I20260806 03:42:32.808418 4613 webserver.cc:460] Webserver started at http://127.30.24.62:39387/ using document root <none> and password file <none>
I20260806 03:42:32.808669 4613 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:32.808739 4613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:32.808895 4613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:32.809950 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "bca90d7c1b9d46bb973116b7e9776a6e"
format_stamp: "Formatted at 2026-08-06 03:42:32 on dist-test-slave-10pc"
server_key: "bae2aae5bbaf887822b5ad1d43b7c26f"
server_key_iv: "2f2ac7b96db674ff3de9578129f79d3c"
server_key_version: "encryptionkey@0"
I20260806 03:42:32.810361 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "bca90d7c1b9d46bb973116b7e9776a6e"
format_stamp: "Formatted at 2026-08-06 03:42:32 on dist-test-slave-10pc"
server_key: "bae2aae5bbaf887822b5ad1d43b7c26f"
server_key_iv: "2f2ac7b96db674ff3de9578129f79d3c"
server_key_version: "encryptionkey@0"
I20260806 03:42:32.811703 4613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:32.812395 4629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:32.812597 4613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:32.812726 4613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "bca90d7c1b9d46bb973116b7e9776a6e"
format_stamp: "Formatted at 2026-08-06 03:42:32 on dist-test-slave-10pc"
server_key: "bae2aae5bbaf887822b5ad1d43b7c26f"
server_key_iv: "2f2ac7b96db674ff3de9578129f79d3c"
server_key_version: "encryptionkey@0"
I20260806 03:42:32.812808 4613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:32.829741 4613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:32.831717 4613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:32.831885 4613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:32.836066 4613 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:42619
I20260806 03:42:32.836103 4682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:42619 every 8 connection(s)
I20260806 03:42:32.836537 4613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:32.837567 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:32.838012 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4613
I20260806 03:42:32.838132 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:32.838328 30816 external_mini_cluster.cc:1428] Setting key 90c880cf9185a252089f8737699de845
I20260806 03:42:32.840657 4683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Bootstrap starting.
I20260806 03:42:32.841274 4683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:32.841586 4683 log.cc:824] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:32.842360 4683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: No bootstrap required, opened a new log
I20260806 03:42:32.844921 4683 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } }
I20260806 03:42:32.845074 4683 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:32.845099 4683 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca90d7c1b9d46bb973116b7e9776a6e, State: Initialized, Role: FOLLOWER
I20260806 03:42:32.845248 4683 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } }
I20260806 03:42:32.845345 4683 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:32.845389 4683 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:32.845439 4683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 0 FOLLOWER]: Advancing to term 1
Aug 06 03:42:32 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:32.846171 4683 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } }
I20260806 03:42:32.846300 4683 leader_election.cc:302] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca90d7c1b9d46bb973116b7e9776a6e; no voters:
I20260806 03:42:32.846482 4683 leader_election.cc:288] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:32.846508 4688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:32.846750 4683 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:32.846995 4688 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 LEADER]: Becoming Leader. State: Replica: bca90d7c1b9d46bb973116b7e9776a6e, State: Running, Role: LEADER
I20260806 03:42:32.847188 4688 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } }
I20260806 03:42:32.848730 4690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca90d7c1b9d46bb973116b7e9776a6e. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } }
I20260806 03:42:32.848842 4690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
W20260806 03:42:32.848928 4703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:42:32.848981 4703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:42:32.849094 4705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:32.849221 4689 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } }
I20260806 03:42:32.849354 4705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:32.849401 4689 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:32.850095 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:32.838901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.30.24.62:42619)
0806 03:42:32.839068 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:32.839077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:32.839117 (+ 40us) server_negotiation.cc:407] Connection header received
0806 03:42:32.839512 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:32.839518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:32.839562 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:32.839686 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:32.840527 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:32.841486 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:32.842541 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:32.842947 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:32.843029 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:32.846001 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:32.846005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:32.846005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:32.846015 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:32.848149 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:32.848646 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:32.848649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:32.848649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:32.848764 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:32.848999 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:32.849000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:32.849001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:32.849309 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:32.849459 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:32.849754 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 03:42:32.849900 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260806 03:42:32.851105 4705 catalog_manager.cc:1347] Generated new cluster ID: 54a42214f7124ac6939f1bfbfc3f817b
I20260806 03:42:32.851166 4705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:32.872402 4705 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:32.872990 4705 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:32.881260 4705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Generated new TSK 0
I20260806 03:42:32.881474 4705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab.
Entry for principal oryx/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab.
I20260806 03:42:32.917618 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42619
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:33.001966 4709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:33.002183 4709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:33.002207 4709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:33.003394 4709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:33.003437 4709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:33.003464 4709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:33.004892 4709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:33.005096 4709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:33.005286 4709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:33.010280 4715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.010332 4717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.010291 4714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.010649 4709 server_base.cc:1034] running on GCE node
I20260806 03:42:33.010860 4709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:33.011113 4709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:33.012274 4709 hybrid_clock.cc:648] HybridClock initialized: now 1785987753012262 us; error 26 us; skew 500 ppm
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:33.014842 4709 init.cc:377] Logged in from keytab as oryx/127.30.24.1@KRBTEST.COM (short username oryx)
I20260806 03:42:33.015296 4709 webserver.cc:460] Webserver started at http://127.30.24.1:40437/ using document root <none> and password file <none>
I20260806 03:42:33.015535 4709 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.015609 4709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.015766 4709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:33.016968 4709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "f0303fb37ba5426c90b4378df948177a"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "e26f20c67bd3612d78848bcb18595d60"
server_key_iv: "5ae6ad8abb5639bcceb0c15b67ba5bbc"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.017402 4709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "f0303fb37ba5426c90b4378df948177a"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "e26f20c67bd3612d78848bcb18595d60"
server_key_iv: "5ae6ad8abb5639bcceb0c15b67ba5bbc"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.018839 4709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:42:33.019668 4724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.019867 4709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:33.019980 4709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "f0303fb37ba5426c90b4378df948177a"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "e26f20c67bd3612d78848bcb18595d60"
server_key_iv: "5ae6ad8abb5639bcceb0c15b67ba5bbc"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.020097 4709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.036055 4709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:33.038156 4709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:33.038332 4709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:33.038594 4709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:33.038940 4709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:33.039006 4709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.039047 4709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:33.039090 4709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.044941 4709 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:43743
I20260806 03:42:33.044988 4837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:43743 every 8 connection(s)
I20260806 03:42:33.045888 4709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:33.053267 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4709
I20260806 03:42:33.053399 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:33.053638 30816 external_mini_cluster.cc:1428] Setting key c8450aec51f94b0752aea1e13273774a
I20260806 03:42:33.056171 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.046632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:48345 (local address 127.30.24.62:42619)
0806 03:42:33.046721 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.046723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.046987 (+ 264us) server_negotiation.cc:407] Connection header received
0806 03:42:33.047492 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.047493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.047504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.047606 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.048717 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.049240 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.050268 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.050487 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.050576 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.052803 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.052804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.052804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.052811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.054691 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.055107 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.055108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.055109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.055183 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.055419 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.055420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.055420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.055570 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.055640 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.055901 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.056019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 03:42:33.056195 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.046605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:48345)
0806 03:42:33.046795 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.046873 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.047401 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.047687 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.047690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.048326 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.048607 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.048630 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.049345 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.049347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.050170 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.050172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.050590 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.050591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.050719 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.051231 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.051240 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.052710 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.054770 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.054777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.054779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.055028 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.055252 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.055253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.055253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.055355 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.055643 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.055648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.055773 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.055856 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.055983 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260806 03:42:33.057036 4838 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42619
I20260806 03:42:33.057168 4838 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:33.057355 4838 heartbeater.cc:511] Master 127.30.24.62:42619 requested a full tablet report, sending...
I20260806 03:42:33.057976 4646 ts_manager.cc:194] Registered new tserver with Master: f0303fb37ba5426c90b4378df948177a (127.30.24.1:43743)
I20260806 03:42:33.058907 4646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:48345
WARNING: no policy specified for oryx/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab.
Entry for principal oryx/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab.
I20260806 03:42:33.084677 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42619
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:33.170312 4843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:33.170569 4843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:33.170610 4843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:33.171816 4843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:33.171888 4843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:33.171932 4843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:33.173481 4843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:33.173768 4843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:33.174036 4843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:33.179468 4852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.179502 4849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.179618 4843 server_base.cc:1034] running on GCE node
W20260806 03:42:33.179805 4850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.180069 4843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:33.180395 4843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:33.181562 4843 hybrid_clock.cc:648] HybridClock initialized: now 1785987753181538 us; error 40 us; skew 500 ppm
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:33.184144 4843 init.cc:377] Logged in from keytab as oryx/127.30.24.2@KRBTEST.COM (short username oryx)
I20260806 03:42:33.184696 4843 webserver.cc:460] Webserver started at http://127.30.24.2:39115/ using document root <none> and password file <none>
I20260806 03:42:33.184940 4843 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.185024 4843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.185169 4843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:33.186280 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "2bf6d272c57a4f42bd5b876560a13e2f"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "5623234fcde48894b16ff3858e64c32d"
server_key_iv: "305578dfa62f37b9cf060eb4bbdb0797"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.186666 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "2bf6d272c57a4f42bd5b876560a13e2f"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "5623234fcde48894b16ff3858e64c32d"
server_key_iv: "305578dfa62f37b9cf060eb4bbdb0797"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.188104 4843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:33.189312 4859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.189535 4843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:33.189652 4843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "2bf6d272c57a4f42bd5b876560a13e2f"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "5623234fcde48894b16ff3858e64c32d"
server_key_iv: "305578dfa62f37b9cf060eb4bbdb0797"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.189749 4843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.233647 4843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:33.235814 4843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:33.236001 4843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:33.236299 4843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:33.236665 4843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:33.236723 4843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.236768 4843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:33.236795 4843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.243037 4843 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:32995
I20260806 03:42:33.243089 4972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:32995 every 8 connection(s)
I20260806 03:42:33.243430 4843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:33.251102 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4843
I20260806 03:42:33.251211 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:33.251418 30816 external_mini_cluster.cc:1428] Setting key 7c090965e7cea2be9b45d9afa44ee907
I20260806 03:42:33.254276 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.244153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:52887 (local address 127.30.24.62:42619)
0806 03:42:33.244235 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.244237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.244447 (+ 210us) server_negotiation.cc:407] Connection header received
0806 03:42:33.244990 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.244991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.245547 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.245615 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.246733 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.247369 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.248477 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.248678 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.248740 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.250827 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.250829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.250829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.250836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.252385 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.253145 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.253146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.253147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.253203 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.253504 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.253505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.253505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.253611 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.253666 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.254002 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.254145 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:42:33.254452 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.244125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:52887)
0806 03:42:33.244327 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.244364 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.244875 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.245703 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.245706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.246412 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.246627 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.246632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.247498 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.247498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.248365 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.248368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.248754 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.248755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.248855 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.249353 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.249361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.250741 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.252665 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.252673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.252675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.253029 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.253278 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.253279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.253279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.253436 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.253681 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.253686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.253860 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.253977 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.254189 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":118,"threads_started":1}
I20260806 03:42:33.255013 4973 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42619
I20260806 03:42:33.255141 4973 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:33.255359 4973 heartbeater.cc:511] Master 127.30.24.62:42619 requested a full tablet report, sending...
I20260806 03:42:33.255795 4646 ts_manager.cc:194] Registered new tserver with Master: 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995)
I20260806 03:42:33.256222 4646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:52887
WARNING: no policy specified for oryx/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab.
Entry for principal oryx/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab.
I20260806 03:42:33.280102 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42619
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:33.364761 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:33.364974 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:33.365015 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:33.366197 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:33.366242 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:33.366297 4978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:33.367758 4978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:33.368002 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:33.368245 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:33.373284 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.373311 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.373371 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.373627 4978 server_base.cc:1034] running on GCE node
I20260806 03:42:33.373859 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:33.374104 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:33.375258 4978 hybrid_clock.cc:648] HybridClock initialized: now 1785987753375237 us; error 36 us; skew 500 ppm
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:33.377847 4978 init.cc:377] Logged in from keytab as oryx/127.30.24.3@KRBTEST.COM (short username oryx)
I20260806 03:42:33.378333 4978 webserver.cc:460] Webserver started at http://127.30.24.3:40789/ using document root <none> and password file <none>
I20260806 03:42:33.378575 4978 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.378623 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.378782 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:33.379942 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "38b7ad462ff847b7b85ed1219d4bccce"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "070279cd4036c11205c45bedceb7518f"
server_key_iv: "43597c5f025b98dfe033dd3e90b8994d"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.380344 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "38b7ad462ff847b7b85ed1219d4bccce"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "070279cd4036c11205c45bedceb7518f"
server_key_iv: "43597c5f025b98dfe033dd3e90b8994d"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.381945 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:33.382784 4993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.382977 4978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:33.383064 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "38b7ad462ff847b7b85ed1219d4bccce"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "070279cd4036c11205c45bedceb7518f"
server_key_iv: "43597c5f025b98dfe033dd3e90b8994d"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.383121 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.398609 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:33.401585 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:33.401768 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:33.402084 4978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:33.402463 4978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:33.402501 4978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.402525 4978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:33.402609 4978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.408878 4978 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:42535
I20260806 03:42:33.408986 5106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:42535 every 8 connection(s)
I20260806 03:42:33.409305 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:33.415453 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 4978
I20260806 03:42:33.415562 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:33.415761 30816 external_mini_cluster.cc:1428] Setting key 2d2853e76a1ceb382fee71c7e49d7ba5
I20260806 03:42:33.419014 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.410073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:36465 (local address 127.30.24.62:42619)
0806 03:42:33.410140 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.410142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.410351 (+ 209us) server_negotiation.cc:407] Connection header received
0806 03:42:33.410873 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.410874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.410886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.410955 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.411947 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.412592 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.413562 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.413762 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.413826 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.416315 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.416316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.416316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.416323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.417756 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.418179 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.418180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.418181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.418234 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.418449 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.418449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.418450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.418548 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.418599 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.418888 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.418950 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:42:33.419302 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.410021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:36465)
0806 03:42:33.410218 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.410264 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.410753 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.411039 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.411042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.411646 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.411846 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.411851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.412675 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.412675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.413469 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.413471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.413842 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.413843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.413945 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.414420 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.414428 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.416203 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.417829 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.417835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.417837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.418094 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.418294 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.418294 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.418294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.418395 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.418628 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.418635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.418761 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.418861 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.418989 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260806 03:42:33.419940 5107 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42619
I20260806 03:42:33.420068 5107 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:33.420258 5107 heartbeater.cc:511] Master 127.30.24.62:42619 requested a full tablet report, sending...
I20260806 03:42:33.420799 4646 ts_manager.cc:194] Registered new tserver with Master: 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535)
I20260806 03:42:33.421177 4646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:36465
I20260806 03:42:33.427853 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.24.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.61.keytab.
Entry for principal oryx/127.30.24.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.61.keytab.
I20260806 03:42:33.453466 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:33613
--webserver_interface=127.30.24.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.61.keytab
--principal=oryx/127.30.24.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:42619,127.30.24.61:33613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:33.540637 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:33.540901 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:33.540943 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:33.542105 5112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:33.542177 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:33.542200 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:33.542235 5112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:33.542268 5112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:33.543756 5112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:42619,127.30.24.61:33613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.61.keytab
--principal=oryx/127.30.24.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:33613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.24.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
W20260806 03:42:33.543907 5112 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.24.62:42619,127.30.24.61:33613'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 03:42:33.544075 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:33.544315 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:33.549392 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.549414 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.549402 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.549679 5112 server_base.cc:1034] running on GCE node
I20260806 03:42:33.549948 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:33.550196 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:33.551340 5112 hybrid_clock.cc:648] HybridClock initialized: now 1785987753551324 us; error 32 us; skew 500 ppm
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:33.553989 5112 init.cc:377] Logged in from keytab as oryx/127.30.24.61@KRBTEST.COM (short username oryx)
I20260806 03:42:33.554499 5112 webserver.cc:460] Webserver started at http://127.30.24.61:36263/ using document root <none> and password file <none>
I20260806 03:42:33.554739 5112 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.554807 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.554963 5112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:33.556077 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/instance:
uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "6979d9543211f1368932ed9673e54e3a"
server_key_iv: "3fb8382e011d535ad4a2ed61532f946b"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.556524 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal/instance:
uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "6979d9543211f1368932ed9673e54e3a"
server_key_iv: "3fb8382e011d535ad4a2ed61532f946b"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.557919 5112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:33.558646 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.558825 5112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.558933 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "6979d9543211f1368932ed9673e54e3a"
server_key_iv: "3fb8382e011d535ad4a2ed61532f946b"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.559013 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.568976 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:33.570959 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:33.571126 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:33.575367 5112 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.61:33613
I20260806 03:42:33.575419 5179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.61:33613 every 8 connection(s)
I20260806 03:42:33.575867 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
I20260806 03:42:33.576938 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:33.578572 5180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:42:33.579532 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 5112
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.61@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
I20260806 03:42:33.589823 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.579325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.30.24.62:42619)
0806 03:42:33.579399 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.579402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.579690 (+ 288us) server_negotiation.cc:407] Connection header received
0806 03:42:33.580366 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.580367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.580381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.580496 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.581318 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.582124 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.583393 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.583695 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.583765 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.586355 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.586357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.586357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.586365 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.588102 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.588727 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.588728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.588729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.588783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.589077 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.589078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.589078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.589243 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.589334 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.589633 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.589725 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:42:33.590014 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.579321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:54774)
0806 03:42:33.579539 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.579598 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.580263 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.580537 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.580558 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.580934 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.581187 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.581194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.582230 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.582231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.583309 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.583311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.583777 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.583778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.583870 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.584415 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.584425 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.586259 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.588203 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.588209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.588212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.588632 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.588852 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.588853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.588853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.589006 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.589325 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.589333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.589481 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.589589 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.589756 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260806 03:42:33.590261 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.580067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.30.24.61:33613)
0806 03:42:33.580204 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.580209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.580245 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:42:33.580364 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.580366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.580398 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.580582 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.581324 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.582453 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.583469 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.583907 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.583957 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.586827 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.586835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.586836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.586843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.588792 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.589243 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.589246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.589247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.589306 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.589525 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.589526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.589527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.589672 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.589856 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.590060 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.590153 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260806 03:42:33.590876 5180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.61@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
I20260806 03:42:33.597055 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.591139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50508)
0806 03:42:33.591235 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.591244 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.591289 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.591391 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.591391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.591498 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.591544 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.591545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.592240 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.592241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.592770 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.592771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.593092 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.593093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.593165 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.593618 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.593623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.594743 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.596079 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.596080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.596081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.596259 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.596529 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.596530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.596530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.596584 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.596794 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.596795 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.596832 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.596883 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.596961 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 03:42:33.597173 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.591193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.30.24.61:33613)
0806 03:42:33.591256 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.591257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.591277 (+ 20us) server_negotiation.cc:407] Connection header received
0806 03:42:33.591325 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.591326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.591335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.591386 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.591625 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.592166 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.592838 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.593027 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.593082 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.594836 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.594837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.594837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.594843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.596005 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.596344 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.596345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.596346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.596406 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.596628 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.596628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.596629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.596739 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.596791 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.596901 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.597012 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":45}
I20260806 03:42:33.598599 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Bootstrap starting.
I20260806 03:42:33.599479 5180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.599819 5180 log.cc:824] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:33.600652 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: No bootstrap required, opened a new log
I20260806 03:42:33.602094 5180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } }
I20260806 03:42:33.602232 5180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.602255 5180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5d0dbf762d407a85a8aaa3a370025e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.602388 5180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } }
I20260806 03:42:33.602679 5186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } } }
I20260806 03:42:33.602741 5180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:33.602813 5186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: FOLLOWER
W20260806 03:42:33.604544 5197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:42:33.604606 5197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:42:33.606453 5112 master_runner.cc:428] Detected that this master 6f5d0dbf762d407a85a8aaa3a370025e is joining an existing cluster
I20260806 03:42:33.606518 5112 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.24.61:33613
I20260806 03:42:33.614320 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.607417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50514)
0806 03:42:33.607662 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.607673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.607745 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.607897 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.607898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.608094 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.608193 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.608195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.608943 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.608944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.609836 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.609839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.610312 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.610313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.610382 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.610934 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.610939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.611409 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.613207 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.613209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.613375 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.613587 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.613588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.613647 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.613890 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.613891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.613944 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.614044 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.614187 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260806 03:42:33.614737 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.607819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:54782)
0806 03:42:33.608319 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.608329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.608376 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.608535 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.608536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.608716 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.608783 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.608785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.609526 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.609527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.610243 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.610244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.610678 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.610679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.610755 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.611137 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.611141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.611579 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.613482 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.613484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.613704 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.613926 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.613927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.613982 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.614265 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.614266 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.614308 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.614397 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.614491 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":85,"threads_started":1}
I20260806 03:42:33.614948 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.607466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.30.24.62:42619)
0806 03:42:33.607518 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.607520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.608390 (+ 870us) server_negotiation.cc:407] Connection header received
0806 03:42:33.608419 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.608421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.608433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.608529 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.608869 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.609444 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.610340 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.610553 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.610597 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.611679 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.611680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.611680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.611689 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.613387 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613796 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.613797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.613798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.613854 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.614049 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.614050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.614050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.614174 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.614217 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.614638 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.614707 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:42:33.614329 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.607596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.30.24.61:33613)
0806 03:42:33.607633 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.607634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.607739 (+ 105us) server_negotiation.cc:407] Connection header received
0806 03:42:33.607804 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.607805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.607816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.607890 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.608270 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.608865 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.609925 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.610213 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.610288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.611499 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.611500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.611501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.611507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.613103 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613461 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.613462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.613462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.613513 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.613710 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.613711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.613711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.613825 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.613900 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.614066 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.614243 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":22}
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.62@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
I20260806 03:42:33.625064 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.617489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.30.24.61:33613)
0806 03:42:33.617531 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.617533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.617702 (+ 169us) server_negotiation.cc:407] Connection header received
0806 03:42:33.617789 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.617790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.617798 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.617839 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.618340 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.618783 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.619905 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.620074 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.620124 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.622292 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.622293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.622293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.622301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.623715 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.624116 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.624118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.624118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.624176 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.624414 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.624415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.624415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.624576 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.624669 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.624865 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.624946 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:33.625077 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.617406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50520)
0806 03:42:33.617574 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.617623 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.617711 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.617855 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.617858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.618206 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.618266 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.618268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.618861 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.618862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.619823 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.619825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.620145 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.620146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.620231 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.620703 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.620709 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.622190 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.623813 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.623821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.623823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.624028 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.624243 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.624244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.624245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.624313 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.624664 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.624673 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.624728 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.624830 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.624949 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260806 03:42:33.625674 4646 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.24.61:33613: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
member_type: NON_VOTER
last_known_addr {
host: "127.30.24.61"
port: 33613
}
attrs {
promote: true
}
}
dest_uuid: "bca90d7c1b9d46bb973116b7e9776a6e"
cas_config_opid_index: -1
I20260806 03:42:33.626037 4646 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } }
I20260806 03:42:33.626567 5207 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6f5d0dbf762d407a85a8aaa3a370025e (127.30.24.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } }
I20260806 03:42:33.626958 5154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.627547 5207 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.24.61:33613
I20260806 03:42:33.627789 5206 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } }
I20260806 03:42:33.627929 5154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Refusing update from remote peer bca90d7c1b9d46bb973116b7e9776a6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 03:42:33.627974 5206 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:33.628016 5112 master.cc:537] Master@127.30.24.61:33613 shutting down...
I20260806 03:42:33.628698 5206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:42:33.629453 4631 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 03:42:33.630031 4631 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e -> Peer 6f5d0dbf762d407a85a8aaa3a370025e (127.30.24.61:33613): Couldn't send request to peer 6f5d0dbf762d407a85a8aaa3a370025e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:33.631523 5112 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:33.631601 5112 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:33.631668 5112 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: stopping tablet replica
I20260806 03:42:33.644915 5112 master.cc:552] Master@127.30.24.61:33613 shutdown complete.
I20260806 03:42:33.644995 5112 master_runner.cc:305] Clearing existing system tablet
I20260806 03:42:33.645145 5112 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.645208 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.645880 5112 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:33.646442 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.646605 5112 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.646692 5112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:33.646720 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "6979d9543211f1368932ed9673e54e3a"
server_key_iv: "3fb8382e011d535ad4a2ed61532f946b"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.647770 5112 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:33.649523 5112 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 03:42:33.649600 5112 log.cc:1196] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 03:42:33.649811 5112 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Deleting consensus metadata
I20260806 03:42:33.649933 5112 master_runner.cc:315] Copying system tablet from 127.30.24.62:42619
I20260806 03:42:33.650556 5112 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: tablet copy: Beginning tablet copy session from remote peer at address 127.30.24.62:42619
I20260806 03:42:33.656889 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.650983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.30.24.62:42619)
0806 03:42:33.651036 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.651039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.651097 (+ 58us) server_negotiation.cc:407] Connection header received
0806 03:42:33.651177 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.651178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.651189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.651263 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.651624 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.652121 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.652958 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.653163 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.653253 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.654245 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.654247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.654247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.654254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.655651 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.656041 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.656044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.656044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.656102 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.656310 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.656312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.656312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.656428 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.656531 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.656713 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.656792 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:42:33.656909 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.650903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:54788)
0806 03:42:33.651028 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.651042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.651117 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.651284 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.651285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.651465 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.651529 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.651530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.652222 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.652224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.652861 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.652863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.653255 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.653256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.653329 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.653854 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.653858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.654152 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.655765 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.655766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.655767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.655949 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.656183 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.656184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.656184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.656246 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.656553 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.656554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.656609 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.656685 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.656794 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260806 03:42:33.657277 4666 tablet_copy_service.cc:140] P bca90d7c1b9d46bb973116b7e9776a6e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6f5d0dbf762d407a85a8aaa3a370025e ({username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:54788)
I20260806 03:42:33.657413 4666 tablet_copy_service.cc:161] P bca90d7c1b9d46bb973116b7e9776a6e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6f5d0dbf762d407a85a8aaa3a370025e at {username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:54788: session id = 6f5d0dbf762d407a85a8aaa3a370025e-00000000000000000000000000000000
I20260806 03:42:33.658141 4666 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:33.658842 5112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 03:42:33.660238 5112 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:33.660390 5112 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:33.661429 5112 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:33.662392 4666 tablet_copy_service.cc:342] P bca90d7c1b9d46bb973116b7e9776a6e: Request end of tablet copy session 6f5d0dbf762d407a85a8aaa3a370025e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:54788
I20260806 03:42:33.662454 4666 tablet_copy_service.cc:433] P bca90d7c1b9d46bb973116b7e9776a6e: ending tablet copy session 6f5d0dbf762d407a85a8aaa3a370025e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6f5d0dbf762d407a85a8aaa3a370025e
I20260806 03:42:33.665309 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:33.666512 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.666651 5218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.666754 5219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.666884 5112 server_base.cc:1034] running on GCE node
I20260806 03:42:33.666961 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:33.667100 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:33.668232 5112 hybrid_clock.cc:648] HybridClock initialized: now 1785987753668233 us; error 33 us; skew 500 ppm
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:33.669675 5112 init.cc:377] Logged in from keytab as oryx/127.30.24.61@KRBTEST.COM (short username oryx)
I20260806 03:42:33.669968 5112 webserver.cc:460] Webserver started at http://127.30.24.61:44805/ using document root <none> and password file <none>
I20260806 03:42:33.670095 5112 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.670145 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.670790 5112 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:33.671339 5228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.671463 5112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.671512 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "6979d9543211f1368932ed9673e54e3a"
server_key_iv: "3fb8382e011d535ad4a2ed61532f946b"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.671594 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.686510 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:33.688190 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:33.691329 5112 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.61:33613
I20260806 03:42:33.691777 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
I20260806 03:42:33.691859 5280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.61:33613 every 8 connection(s)
I20260806 03:42:33.692931 5281 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:42:33.693493 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Bootstrap starting.
I20260806 03:42:33.696357 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:33.696623 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Bootstrap complete.
I20260806 03:42:33.696842 5281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } }
I20260806 03:42:33.696936 5281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f5d0dbf762d407a85a8aaa3a370025e, State: Initialized, Role: LEARNER
I20260806 03:42:33.696982 5281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } }
I20260806 03:42:33.697160 5281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:33.697173 5285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } }
I20260806 03:42:33.697407 5285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: LEARNER
I20260806 03:42:33.698922 5296 catalog_manager.cc:1259] Loaded cluster ID: 54a42214f7124ac6939f1bfbfc3f817b
I20260806 03:42:33.698997 5296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: loading cluster ID for follower catalog manager: success
I20260806 03:42:33.700304 5296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: acquiring CA information for follower catalog manager: success
I20260806 03:42:33.700630 5296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:42:33.701104 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.691977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.30.24.61:33613)
0806 03:42:33.692115 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.692118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.692127 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:33.692157 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.692158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.692169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.692246 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.693112 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.694210 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.695229 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.695546 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.695625 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.697186 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.697187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.697187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.697195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.699055 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.699950 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.699952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.699952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.700020 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.700251 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.700252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.700252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.700385 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.700507 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.700919 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.701014 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.30.24.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.60.keytab.
Entry for principal oryx/127.30.24.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.60.keytab.
I20260806 03:42:33.735440 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:43969
--webserver_interface=127.30.24.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.60.keytab
--principal=oryx/127.30.24.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:33.830371 5299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:33.830569 5299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:33.830591 5299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:33.831748 5299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:33.831787 5299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:33.831801 5299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:33.831815 5299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:33.831826 5299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:33.833299 5299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.60.keytab
--principal=oryx/127.30.24.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.24.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:33.833575 5299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:33.833806 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:33.838693 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.838801 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:33.838693 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:33.838975 5299 server_base.cc:1034] running on GCE node
I20260806 03:42:33.839236 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:33.839521 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:33.840682 5299 hybrid_clock.cc:648] HybridClock initialized: now 1785987753840672 us; error 51 us; skew 500 ppm
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:33.843106 5299 init.cc:377] Logged in from keytab as oryx/127.30.24.60@KRBTEST.COM (short username oryx)
I20260806 03:42:33.843535 5299 webserver.cc:460] Webserver started at http://127.30.24.60:45953/ using document root <none> and password file <none>
I20260806 03:42:33.843776 5299 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.843824 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.843981 5299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:33.845134 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/instance:
uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "311340356cb25585130486c181289c6d"
server_key_iv: "e4d83a11fee615f8de80dd01e53b5fba"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.845507 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal/instance:
uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "311340356cb25585130486c181289c6d"
server_key_iv: "e4d83a11fee615f8de80dd01e53b5fba"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.846891 5299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:33.847667 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.847855 5299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:33.847954 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "311340356cb25585130486c181289c6d"
server_key_iv: "e4d83a11fee615f8de80dd01e53b5fba"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.848030 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.882457 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:33.884948 5299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:33.885131 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:33.889303 5299 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.60:43969
I20260806 03:42:33.889364 5366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.60:43969 every 8 connection(s)
I20260806 03:42:33.889725 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
I20260806 03:42:33.890753 5367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:33.892299 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 5299
I20260806 03:42:33.892338 5367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.60@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.24.60@KRBTEST.COM
I20260806 03:42:33.903262 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.893337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.30.24.62:42619)
0806 03:42:33.893384 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.893386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.893472 (+ 86us) server_negotiation.cc:407] Connection header received
0806 03:42:33.893896 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.893898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.893912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.893971 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.894733 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.895526 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.896828 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.897205 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.897292 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.899964 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.899965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.899966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.899972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.901686 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.902116 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.902117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.902118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.902184 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.902486 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.902487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.902487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.902660 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.902731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.903060 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.903150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:42:33.903439 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.893225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:54790)
0806 03:42:33.893354 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.893391 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.893788 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.894014 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.894017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.894402 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.894645 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.894650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.895628 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.895629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.896662 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.896664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.897337 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.897339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.897490 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.898258 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.898273 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.899875 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.901797 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.901804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.901807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.902035 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.902260 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.902261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.902262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.902399 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.902736 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.902744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.902911 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.903007 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.903180 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260806 03:42:33.903455 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.893029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.30.24.60:43969)
0806 03:42:33.893213 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.893222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.893258 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:42:33.893774 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.893781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.893802 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.893929 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.894708 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.895844 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.896845 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.897385 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.897490 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.900836 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.900843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.900844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.900849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.902239 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.902596 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.902599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.902600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.902653 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.902871 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.902871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.902872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.903028 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.903176 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.903261 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.903339 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260806 03:42:33.904325 5367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.60@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
I20260806 03:42:33.910626 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.904666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50558)
0806 03:42:33.904710 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.904720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.904757 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.904907 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.904908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.905045 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.905090 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.905091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.905700 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.905701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.906194 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.906195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.906512 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.906512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.906568 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.906901 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.906905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.908027 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.909681 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.909682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.909682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.909867 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.910103 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.910104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.910104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.910168 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.910378 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.910379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.910416 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.910481 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.910557 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 03:42:33.910636 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.904743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.30.24.61:33613)
0806 03:42:33.904796 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.904798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.904805 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:42:33.904828 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.904829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.904838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.904885 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.905163 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.905636 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.906262 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.906451 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.906499 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.908125 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.908127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.908127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.908134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.909601 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.909960 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.909960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.909961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.910025 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.910223 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.910224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.910224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.910323 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.910372 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.910503 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.910570 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:33.911162 5367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.60@KRBTEST.COM for oryx/127.30.24.60@KRBTEST.COM
I20260806 03:42:33.917667 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.911437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51166)
0806 03:42:33.911490 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.911501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.911568 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.911674 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.911674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.911790 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.911820 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.911821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.912401 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.912402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.912978 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.912979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.913337 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.913338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.913382 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.913788 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.913793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.914871 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.916680 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.916681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.916681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.916843 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.917039 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.917040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.917040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.917093 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.917349 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.917349 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.917386 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.917454 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.917548 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 03:42:33.917675 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.911485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.30.24.60:43969)
0806 03:42:33.911520 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.911521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.911548 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:42:33.911618 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.911619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.911627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.911670 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.911876 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.912328 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.913048 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.913255 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.913308 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.914940 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.914941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.914942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.914947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.916572 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.916915 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.916916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.916917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.916967 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.917165 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.917166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.917167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.917284 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.917356 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.917483 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.917595 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":16}
I20260806 03:42:33.919150 5367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Bootstrap starting.
I20260806 03:42:33.919945 5367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.920205 5367 log.cc:824] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:33.920967 5367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: No bootstrap required, opened a new log
I20260806 03:42:33.922387 5367 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } }
I20260806 03:42:33.922523 5367 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.922545 5367 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ebec280f1f4fd382f5b2a482dfd5b5, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.922690 5367 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } }
I20260806 03:42:33.922983 5372 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } } }
I20260806 03:42:33.923080 5367 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:33.923084 5372 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 03:42:33.924841 5383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:42:33.924901 5383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:42:33.927103 5299 master_runner.cc:428] Detected that this master c9ebec280f1f4fd382f5b2a482dfd5b5 is joining an existing cluster
I20260806 03:42:33.927162 5299 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.24.60:43969
I20260806 03:42:33.935206 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.928603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.30.24.60:43969)
0806 03:42:33.928639 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.928641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.928735 (+ 94us) server_negotiation.cc:407] Connection header received
0806 03:42:33.928812 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.928813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.928820 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.928844 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.929277 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.929942 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931176 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.931453 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931492 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.932586 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.932587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.932588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.932595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.934093 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.934443 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.934444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.934445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.934506 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.934693 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.934694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.934694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.934824 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.934898 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.935005 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.935111 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":19}
I20260806 03:42:33.935398 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.928419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51180)
0806 03:42:33.928686 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.928694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.928754 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.928895 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.928896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.929150 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.929205 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.929206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.930028 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.930029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.931074 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.931077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.931577 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.931631 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.932101 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.932124 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.932500 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.934189 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.934190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.934190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.934368 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.934576 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.934576 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.934577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.934631 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.934888 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.934889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.934923 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.934969 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.935049 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":121,"thread_start_us":99,"threads_started":1}
I20260806 03:42:33.953078 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.928381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.30.24.61:33613)
0806 03:42:33.928420 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.928422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.928625 (+ 203us) server_negotiation.cc:407] Connection header received
0806 03:42:33.928730 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.928731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.928742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.928772 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.929281 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.930022 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931205 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.931505 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931545 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.949915 (+ 18370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.949917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.949917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.949925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.951748 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952172 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.952174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.952174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.952237 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952439 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.952440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.952440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.952628 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.952677 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.952886 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.952974 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:42:33.953119 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.928195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.30.24.62:42619)
0806 03:42:33.928240 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.928243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.928377 (+ 134us) server_negotiation.cc:407] Connection header received
0806 03:42:33.928516 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.928517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.928530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.928562 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.929136 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.929946 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931178 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.931471 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931508 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.949877 (+ 18369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.949879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.949879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.949890 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.951923 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952292 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.952293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.952294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.952351 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952571 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.952571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.952572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.952696 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.952774 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.952936 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.953012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:33.953653 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.928203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50564)
0806 03:42:33.928562 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.928572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.928655 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.928835 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.928835 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.929112 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.929182 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.929183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.930133 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.930133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.931087 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.931090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.931625 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.931697 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.949112 (+ 17415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.949119 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.949776 (+ 657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.951843 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.951844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.951845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.952060 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.952314 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.952316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.952372 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.952727 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.952728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.952777 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.953225 (+ 448us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.953336 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":57,"thread_start_us":75,"threads_started":1}
I20260806 03:42:33.953665 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.928179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:54798)
0806 03:42:33.928313 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.928325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.928408 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.928648 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.928649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.928914 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.929040 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.929042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.930028 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.930029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.931077 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.931078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.931580 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.931580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.931647 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.949196 (+ 17549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.949201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.949740 (+ 539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.952009 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.952011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.952203 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.952427 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.952427 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.952428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.952508 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.952766 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.952767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.952812 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.952894 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.953012 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":26,"thread_start_us":80,"threads_started":1}
I20260806 03:42:33.954754 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4709
Aug 06 03:42:33 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987752, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.62@KRBTEST.COM for oryx/127.30.24.60@KRBTEST.COM
I20260806 03:42:33.964031 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:43743
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:42619
--webserver_port=40437
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:42:33.967079 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.959213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51186)
0806 03:42:33.959255 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.959265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.959307 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.959463 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.959464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.959646 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.959702 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.959704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.960658 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.960659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.961521 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.961523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.962022 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.962024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.962115 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.962528 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.962533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:33.964176 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:33.966015 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.966016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.966016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.966240 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.966440 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:33.966441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:33.966441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:33.966496 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:33.966768 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:33.966769 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:33.966813 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.966874 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.966976 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 03:42:33.967427 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.959284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.30.24.60:43969)
0806 03:42:33.959326 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.959329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.959336 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:42:33.959372 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.959372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.959383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.959436 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.959803 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.960565 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.961642 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.961940 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.962005 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.964273 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:33.964275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:33.964275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:33.964282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:33.965920 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.966320 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.966321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.966321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.966376 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:33.966557 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:33.966558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:33.966558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:33.966682 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:33.966741 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.967260 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.967336 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:33.968014 4646 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.24.60:43969: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
member_type: NON_VOTER
last_known_addr {
host: "127.30.24.60"
port: 43969
}
attrs {
promote: true
}
}
dest_uuid: "bca90d7c1b9d46bb973116b7e9776a6e"
cas_config_opid_index: 5
I20260806 03:42:33.968322 4646 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } }
I20260806 03:42:33.968762 5206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.969936 5206 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.24.60:43969
I20260806 03:42:33.970016 5206 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.970130 5206 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:33.970709 5341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.971890 5341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer bca90d7c1b9d46bb973116b7e9776a6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 03:42:33.972186 5299 master.cc:537] Master@127.30.24.60:43969 shutting down...
I20260806 03:42:33.972301 5207 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:42:33.973133 4632 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e -> Peer c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60:43969): Couldn't send request to peer c9ebec280f1f4fd382f5b2a482dfd5b5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:33.975874 5299 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:33.975977 5299 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:33.976038 5299 tablet_replica.cc:330] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: stopping tablet replica
I20260806 03:42:33.976117 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.970248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.30.24.61:33613)
0806 03:42:33.970299 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.970302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.970311 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:33.970339 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.970341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.970366 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.970443 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:33.970635 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.971412 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.974921 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.975521 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.975593 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.975679 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:33.976013 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 03:42:33.976053 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:33.976438 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.968862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50570)
0806 03:42:33.969032 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.969042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.969083 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.970513 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.970514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.970516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:33.970559 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.970560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.971547 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.971548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.974817 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.974820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.975715 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.975716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.975781 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.975804 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 03:42:33.975855 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 03:42:33.975897 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260806 03:42:33.976828 5255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 LEARNER]: Refusing update from remote peer bca90d7c1b9d46bb973116b7e9776a6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 03:42:33.977029 5207 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:42:33.977676 5255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.978890 5285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca90d7c1b9d46bb973116b7e9776a6e. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.978981 5285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: LEARNER
I20260806 03:42:33.979012 5285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: NON_VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: true } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.979050 5285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: LEARNER
I20260806 03:42:33.979677 5206 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: attempting to promote NON_VOTER 6f5d0dbf762d407a85a8aaa3a370025e to VOTER
I20260806 03:42:33.979882 5206 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } }
I20260806 03:42:33.980420 5255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 LEARNER]: Refusing update from remote peer bca90d7c1b9d46bb973116b7e9776a6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 03:42:33.980660 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.980254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51202)
0806 03:42:33.980590 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.980636 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
I20260806 03:42:33.980813 5395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:42:33.981019 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.980948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51212)
0806 03:42:33.980979 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.980999 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260806 03:42:33.981079 4632 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e -> Peer c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60:43969): Couldn't send request to peer c9ebec280f1f4fd382f5b2a482dfd5b5. Status: Network error: Client connection negotiation failed: client connection to 127.30.24.60:43969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:33.981531 5207 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6f5d0dbf762d407a85a8aaa3a370025e (127.30.24.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.981657 5255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6f5d0dbf762d407a85a8aaa3a370025e (127.30.24.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.982440 5207 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.982510 5207 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:33.982555 5207 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.982602 5207 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:33.982786 5394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:33.982926 5394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:42:33.990603 5299 master.cc:552] Master@127.30.24.60:43969 shutdown complete.
I20260806 03:42:33.990662 5299 master_runner.cc:305] Clearing existing system tablet
I20260806 03:42:33.990796 5299 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:33.990846 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:33.991604 5299 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:33.992190 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:33.992340 5299 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:33.992403 5299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:42:33.992424 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "311340356cb25585130486c181289c6d"
server_key_iv: "e4d83a11fee615f8de80dd01e53b5fba"
server_key_version: "encryptionkey@0"
I20260806 03:42:33.993258 5299 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:33.994735 5299 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 03:42:33.994808 5299 log.cc:1196] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 03:42:33.995009 5299 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Deleting consensus metadata
I20260806 03:42:33.995141 5299 master_runner.cc:315] Copying system tablet from 127.30.24.62:42619
I20260806 03:42:33.995741 5299 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.24.62:42619
I20260806 03:42:34.003603 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.996143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.30.24.62:42619)
0806 03:42:33.996190 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:33.996192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:33.996257 (+ 65us) server_negotiation.cc:407] Connection header received
0806 03:42:33.996329 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:33.996330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:33.996341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:33.996415 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:33.996813 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.997630 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.998814 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.999135 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.999205 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.000338 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.000339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.000340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.000348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.002285 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.002731 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.002732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.002733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.002799 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.003024 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.003025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.003025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.003170 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.003250 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.003427 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.003508 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:42:34.003911 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:33.996051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:54810)
0806 03:42:33.996196 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:33.996207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:33.996272 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:33.996436 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:33.996437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:33.996669 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:33.996720 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.996723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.997718 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.997720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.998723 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:33.998726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:33.999217 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:33.999219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:33.999287 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:33.999820 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:33.999825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.000236 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.002416 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.002417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.002418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.002649 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.002876 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.002877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.002877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.002953 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.003246 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.003247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.003297 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.003376 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.003495 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260806 03:42:34.004382 4666 tablet_copy_service.cc:140] P bca90d7c1b9d46bb973116b7e9776a6e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c9ebec280f1f4fd382f5b2a482dfd5b5 ({username='oryx', principal='oryx/127.30.24.60@KRBTEST.COM'} at 127.0.0.1:54810)
I20260806 03:42:34.004477 4666 tablet_copy_service.cc:161] P bca90d7c1b9d46bb973116b7e9776a6e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c9ebec280f1f4fd382f5b2a482dfd5b5 at {username='oryx', principal='oryx/127.30.24.60@KRBTEST.COM'} at 127.0.0.1:54810: session id = c9ebec280f1f4fd382f5b2a482dfd5b5-00000000000000000000000000000000
I20260806 03:42:34.005158 4666 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:34.005728 5299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 03:42:34.007202 5299 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:34.007350 5299 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:34.008235 5299 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:34.009255 4666 tablet_copy_service.cc:342] P bca90d7c1b9d46bb973116b7e9776a6e: Request end of tablet copy session c9ebec280f1f4fd382f5b2a482dfd5b5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.24.60@KRBTEST.COM'} at 127.0.0.1:54810
I20260806 03:42:34.009317 4666 tablet_copy_service.cc:433] P bca90d7c1b9d46bb973116b7e9776a6e: ending tablet copy session c9ebec280f1f4fd382f5b2a482dfd5b5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c9ebec280f1f4fd382f5b2a482dfd5b5
I20260806 03:42:34.012120 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:34.013305 5405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.013684 5408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.016965 5406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:34.017537 5299 server_base.cc:1034] running on GCE node
I20260806 03:42:34.017639 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:34.017782 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:34.018903 5299 hybrid_clock.cc:648] HybridClock initialized: now 1785987754018895 us; error 27 us; skew 500 ppm
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:34.020533 5299 init.cc:377] Logged in from keytab as oryx/127.30.24.60@KRBTEST.COM (short username oryx)
I20260806 03:42:34.020807 5299 webserver.cc:460] Webserver started at http://127.30.24.60:35945/ using document root <none> and password file <none>
I20260806 03:42:34.020925 5299 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:34.020959 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:34.021804 5299 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:34.022426 5415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.022609 5299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:34.022663 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "311340356cb25585130486c181289c6d"
server_key_iv: "e4d83a11fee615f8de80dd01e53b5fba"
server_key_version: "encryptionkey@0"
I20260806 03:42:34.022724 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:34.038313 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:34.040386 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:34.043489 5299 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.60:43969
I20260806 03:42:34.043706 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.60:43969 every 8 connection(s)
I20260806 03:42:34.043836 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
I20260806 03:42:34.044782 5469 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:42:34.045359 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Bootstrap starting.
I20260806 03:42:34.048492 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:34.048717 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: Bootstrap complete.
I20260806 03:42:34.048938 5469 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } }
I20260806 03:42:34.049036 5469 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9ebec280f1f4fd382f5b2a482dfd5b5, State: Initialized, Role: LEARNER
I20260806 03:42:34.049082 5469 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } }
I20260806 03:42:34.049300 5469 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:34.050954 5483 catalog_manager.cc:1259] Loaded cluster ID: 54a42214f7124ac6939f1bfbfc3f817b
I20260806 03:42:34.051061 5483 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: loading cluster ID for follower catalog manager: success
I20260806 03:42:34.052296 5483 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: acquiring CA information for follower catalog manager: success
I20260806 03:42:34.052654 5483 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:42:34.056576 5472 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:34.056699 5472 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: This master's current role is: LEARNER
W20260806 03:42:34.070155 5392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:34.070366 5392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:34.070391 5392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:34.071655 5392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:34.071700 5392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:34.071727 5392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:34.073482 5392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:43743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=40437
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:34.073755 5392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:34.073990 5392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:34.079152 5488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.079152 5489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.079347 5491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:34.079547 5392 server_base.cc:1034] running on GCE node
I20260806 03:42:34.079769 5392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:34.080018 5392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:34.081179 5392 hybrid_clock.cc:648] HybridClock initialized: now 1785987754081168 us; error 34 us; skew 500 ppm
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:34.083870 5392 init.cc:377] Logged in from keytab as oryx/127.30.24.1@KRBTEST.COM (short username oryx)
I20260806 03:42:34.084381 5392 webserver.cc:460] Webserver started at http://127.30.24.1:40437/ using document root <none> and password file <none>
I20260806 03:42:34.084659 5392 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:34.084735 5392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:34.086506 5392 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:34.087076 5498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.087294 5392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:34.087393 5392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "f0303fb37ba5426c90b4378df948177a"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "e26f20c67bd3612d78848bcb18595d60"
server_key_iv: "5ae6ad8abb5639bcceb0c15b67ba5bbc"
server_key_version: "encryptionkey@0"
I20260806 03:42:34.087778 5392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:34.107985 5392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:34.110564 5392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:34.110769 5392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:34.111034 5392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:34.111378 5392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:34.111434 5392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.111470 5392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:34.111497 5392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.117712 5392 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:43743
I20260806 03:42:34.117797 5613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:43743 every 8 connection(s)
I20260806 03:42:34.118162 5392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:34.120270 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 5392
I20260806 03:42:34.120443 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4843
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.60@KRBTEST.COM
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:34.131779 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.119481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:35183 (local address 127.30.24.61:33613)
0806 03:42:34.119528 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.119531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.119584 (+ 53us) server_negotiation.cc:407] Connection header received
0806 03:42:34.119999 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.120001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.120015 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.120049 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.122559 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.123290 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124285 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.124599 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124660 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.127532 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.127534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.127535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.127543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.129113 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.129636 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.129638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.129638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.129715 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.130267 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.130268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.130269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.130436 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.130898 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.131566 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.131669 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:42:34.132495 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.119399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.30.24.1:35183)
0806 03:42:34.119526 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.119537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.119896 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.120581 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.120582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.122393 (+ 1811us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.122452 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.122454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.123392 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.123393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.124183 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.124186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.124754 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.124818 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.125313 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.125318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.127429 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.129207 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.129216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.129219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.129550 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.130028 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.130029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.130030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.130180 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.130957 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.130965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.131385 (+ 420us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.131995 (+ 610us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.132193 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260806 03:42:34.133203 5616 heartbeater.cc:347] Connected to a master server at 127.30.24.61:33613
I20260806 03:42:34.133314 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.118946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:36125 (local address 127.30.24.60:43969)
0806 03:42:34.119094 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.119097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.119405 (+ 308us) server_negotiation.cc:407] Connection header received
0806 03:42:34.120053 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.120054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.120068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.120142 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.121116 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.122069 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.123860 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.124195 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124253 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.129518 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.129520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.129520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.129528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.131581 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132036 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.132037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.132037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.132097 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132309 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.132310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.132311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.132442 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.132531 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.133090 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.133197 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260806 03:42:34.133319 5616 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.132912 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.118947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.30.24.1:36125)
0806 03:42:34.119158 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.119207 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.119899 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.120212 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.120214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.120748 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.121013 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.121018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.122240 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.122241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.123768 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.123771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.124323 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.124481 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.125442 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.125446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.129416 (+ 3970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.131664 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.131665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.131665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.131951 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.132183 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.132184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.132242 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.132567 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.132568 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.132623 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.132710 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.132807 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260806 03:42:34.133517 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.119200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:36305 (local address 127.30.24.62:42619)
0806 03:42:34.119272 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.119274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.119290 (+ 16us) server_negotiation.cc:407] Connection header received
0806 03:42:34.120007 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.120009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.120021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.120075 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.121465 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.122404 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.123893 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.124291 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124341 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.130188 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.130189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.130189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.130198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.131922 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132515 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.132516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.132517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.132584 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132931 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.132932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.132933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.133078 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.133144 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.133347 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.133430 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 03:42:34.133524 5616 heartbeater.cc:511] Master 127.30.24.61:33613 requested a full tablet report, sending...
I20260806 03:42:34.133769 5614 heartbeater.cc:347] Connected to a master server at 127.30.24.60:43969
I20260806 03:42:34.133838 5614 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.133915 5614 heartbeater.cc:511] Master 127.30.24.60:43969 requested a full tablet report, sending...
I20260806 03:42:34.133939 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.118947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:36305)
0806 03:42:34.119214 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.119227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.119875 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.120160 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.120165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.121247 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.121347 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.121351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.122514 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.122515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.123774 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.123778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.124433 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.124434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.124570 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.125161 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.125173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.130075 (+ 4902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.132155 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.132157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.132396 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.132785 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.132785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.132785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.132855 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.133179 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.133180 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.133226 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.133595 (+ 369us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.133699 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":3,"spinlock_wait_cycles":7040,"thread_start_us":103,"threads_started":1}
I20260806 03:42:34.134259 5615 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42619
I20260806 03:42:34.134315 5615 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.134382 5615 heartbeater.cc:511] Master 127.30.24.62:42619 requested a full tablet report, sending...
I20260806 03:42:34.134367 5433 ts_manager.cc:194] Registered new tserver with Master: f0303fb37ba5426c90b4378df948177a (127.30.24.1:43743)
I20260806 03:42:34.134631 4646 ts_manager.cc:194] Re-registered known tserver with Master: f0303fb37ba5426c90b4378df948177a (127.30.24.1:43743)
I20260806 03:42:34.134658 5245 ts_manager.cc:194] Registered new tserver with Master: f0303fb37ba5426c90b4378df948177a (127.30.24.1:43743)
I20260806 03:42:34.135140 4646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:36305
I20260806 03:42:34.135509 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:32995
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:42619
--webserver_port=39115
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:34.220878 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:34.221132 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:34.221175 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:34.222383 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:34.222451 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:34.222498 5622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:34.224097 5622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:32995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=39115
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:34.224385 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:34.224643 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:34.229688 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.229697 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.229698 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:34.230048 5622 server_base.cc:1034] running on GCE node
I20260806 03:42:34.230268 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:34.230468 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:34.231608 5622 hybrid_clock.cc:648] HybridClock initialized: now 1785987754231594 us; error 29 us; skew 500 ppm
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:34.234102 5622 init.cc:377] Logged in from keytab as oryx/127.30.24.2@KRBTEST.COM (short username oryx)
I20260806 03:42:34.234598 5622 webserver.cc:460] Webserver started at http://127.30.24.2:39115/ using document root <none> and password file <none>
I20260806 03:42:34.234841 5622 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:34.234925 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:34.236275 5622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:34.236930 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.237102 5622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.237211 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "2bf6d272c57a4f42bd5b876560a13e2f"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "5623234fcde48894b16ff3858e64c32d"
server_key_iv: "305578dfa62f37b9cf060eb4bbdb0797"
server_key_version: "encryptionkey@0"
I20260806 03:42:34.237743 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:34.253592 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:34.255543 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:34.255709 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:34.255952 5622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:34.256279 5622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:34.256333 5622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.256379 5622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:34.256412 5622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.262418 5622 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:32995
I20260806 03:42:34.262470 5750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:32995 every 8 connection(s)
I20260806 03:42:34.262820 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.60@KRBTEST.COM
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:34.270926 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 5622
I20260806 03:42:34.271096 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4978
I20260806 03:42:34.274855 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.263554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:45903 (local address 127.30.24.60:43969)
0806 03:42:34.264067 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.264069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.264157 (+ 88us) server_negotiation.cc:407] Connection header received
0806 03:42:34.264508 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.264509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.264523 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.264574 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.265476 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.265990 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267011 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.267249 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267335 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.269906 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.269908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.269908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.269917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.272425 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273835 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.273836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.273836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.273896 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.274099 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.274100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.274100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.274237 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.274279 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.274665 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.274760 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
I20260806 03:42:34.274879 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.263556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:33869 (local address 127.30.24.62:42619)
0806 03:42:34.263602 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.263605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.263859 (+ 254us) server_negotiation.cc:407] Connection header received
0806 03:42:34.264503 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.264505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.264519 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.264577 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.265489 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.266249 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267398 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.267618 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267651 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.271012 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.271014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.271014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.271022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.273028 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273646 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.273647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.273648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.273712 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.274060 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.274061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.274061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.274201 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.274250 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.274673 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.274758 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:42:34.274953 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.264013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:44845 (local address 127.30.24.61:33613)
0806 03:42:34.264069 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.264071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.264640 (+ 569us) server_negotiation.cc:407] Connection header received
0806 03:42:34.264701 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.264702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.264716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.264773 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.265489 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.266256 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267403 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.267709 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267775 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.270448 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.270450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.270450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.270459 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.273251 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273856 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.273857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.273857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.273928 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.274186 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.274187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.274188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.274355 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.274407 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.274695 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.274803 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:42:34.275565 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.263559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.30.24.2:45903)
0806 03:42:34.264101 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.264111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.264389 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.264653 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.264659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.265153 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.265374 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.265379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.266101 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.266102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.266915 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.266917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.267348 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.267492 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.268088 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.268096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.269813 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.273344 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.273345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.273745 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.273969 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.273970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.274025 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.274326 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.274327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.274481 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.275002 (+ 521us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.275339 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":129,"spinlock_wait_cycles":7552,"thread_start_us":108,"threads_started":1}
I20260806 03:42:34.275570 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.263924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.30.24.2:44845)
0806 03:42:34.264581 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.264592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.264647 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.264784 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.264785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.265153 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.265375 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.265382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.266352 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.266353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.267294 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.267295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.267774 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.267845 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.268419 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.268424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.270371 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.273337 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.273338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.273745 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.274017 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.274018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.274018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.274096 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.274462 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.274463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.274519 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.275038 (+ 519us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.275261 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":47,"thread_start_us":77,"threads_started":1}
I20260806 03:42:34.276152 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.263559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:33869)
0806 03:42:34.263732 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.263769 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.264372 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.264653 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.264659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.265153 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.265375 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.265382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.266352 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.266353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.267291 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.267293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.267757 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.267758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.267825 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.268330 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.268335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.270909 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.273131 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.273143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.273540 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.273817 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.273818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.273819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.273977 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.274302 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.274310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.274481 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.275519 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.275615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":76,"thread_start_us":107,"threads_started":1}
I20260806 03:42:34.276212 5753 heartbeater.cc:347] Connected to a master server at 127.30.24.61:33613
I20260806 03:42:34.276396 5751 heartbeater.cc:347] Connected to a master server at 127.30.24.60:43969
I20260806 03:42:34.276500 5753 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.276727 5753 heartbeater.cc:511] Master 127.30.24.61:33613 requested a full tablet report, sending...
I20260806 03:42:34.276500 5751 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.276995 5751 heartbeater.cc:511] Master 127.30.24.60:43969 requested a full tablet report, sending...
I20260806 03:42:34.277242 5245 ts_manager.cc:194] Registered new tserver with Master: 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995)
I20260806 03:42:34.277531 5433 ts_manager.cc:194] Registered new tserver with Master: 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995)
I20260806 03:42:34.281196 5752 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42619
I20260806 03:42:34.281273 5752 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.281343 5752 heartbeater.cc:511] Master 127.30.24.62:42619 requested a full tablet report, sending...
I20260806 03:42:34.281639 4646 ts_manager.cc:194] Re-registered known tserver with Master: 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995)
I20260806 03:42:34.282163 4646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:33869
I20260806 03:42:34.284313 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:42535
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:42619
--webserver_port=40789
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:34.369197 5758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:34.369427 5758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:34.369465 5758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:34.370682 5758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:34.370751 5758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:34.370797 5758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:34.372414 5758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:42535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=40789
--tserver_master_addrs=127.30.24.62:42619,127.30.24.61:33613,127.30.24.60:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:34.372691 5758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:34.372946 5758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:34.378036 5764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.378046 5766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:34.378046 5763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:34.378506 5758 server_base.cc:1034] running on GCE node
I20260806 03:42:34.378722 5758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:34.378993 5758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:34.380162 5758 hybrid_clock.cc:648] HybridClock initialized: now 1785987754380155 us; error 36 us; skew 500 ppm
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:34.382728 5758 init.cc:377] Logged in from keytab as oryx/127.30.24.3@KRBTEST.COM (short username oryx)
I20260806 03:42:34.383173 5758 webserver.cc:460] Webserver started at http://127.30.24.3:40789/ using document root <none> and password file <none>
I20260806 03:42:34.383430 5758 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:34.383518 5758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:34.384987 5758 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:34.385632 5773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.385803 5758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.385908 5758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "38b7ad462ff847b7b85ed1219d4bccce"
format_stamp: "Formatted at 2026-08-06 03:42:33 on dist-test-slave-10pc"
server_key: "070279cd4036c11205c45bedceb7518f"
server_key_iv: "43597c5f025b98dfe033dd3e90b8994d"
server_key_version: "encryptionkey@0"
I20260806 03:42:34.386277 5758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:34.397755 5758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:34.399986 5758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:34.400162 5758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:34.400416 5758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:34.400771 5758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:34.400828 5758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.400872 5758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:34.400902 5758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:34.407573 5758 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:42535
I20260806 03:42:34.407626 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:42535 every 8 connection(s)
I20260806 03:42:34.408248 5758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:34.409655 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 5758
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.60@KRBTEST.COM
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.61@KRBTEST.COM
Aug 06 03:42:34 dist-test-slave-10pc krb5kdc[4600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:34.423571 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.409252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:40409 (local address 127.30.24.60:43969)
0806 03:42:34.409537 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.409539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.410777 (+ 1238us) server_negotiation.cc:407] Connection header received
0806 03:42:34.410905 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.410906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.410920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.410957 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.412233 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.413071 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.414742 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.415098 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.415140 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.419632 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.419634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.419634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.419642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.421659 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.422268 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.422270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.422270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.422342 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.422671 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.422673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.422673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.422825 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.422876 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.423342 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.423436 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260806 03:42:34.424273 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.409197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.30.24.3:40409)
0806 03:42:34.410703 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.410715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.410812 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.411155 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.411160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.412068 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.412140 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.412141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.413170 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.413171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.414618 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.414621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.415253 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.415254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.415418 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.416238 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.416309 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.419503 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.421770 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.421778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.421781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.422164 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.422426 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.422426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.422427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.422582 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.422933 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.422940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.423151 (+ 211us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.423742 (+ 591us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.423954 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"mutex_wait_us":65,"thread_start_us":113,"threads_started":1}
I20260806 03:42:34.424831 5887 heartbeater.cc:347] Connected to a master server at 127.30.24.60:43969
I20260806 03:42:34.424942 5887 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.425172 5887 heartbeater.cc:511] Master 127.30.24.60:43969 requested a full tablet report, sending...
I20260806 03:42:34.425293 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.411097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.30.24.62:42619)
0806 03:42:34.412273 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.412275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.412283 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:34.412307 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.412308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.412320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.413221 (+ 901us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.413264 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.414091 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.416314 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.416687 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.416802 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.418289 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.418291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.418291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.418299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.422835 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.423932 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.423934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.423934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.424001 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.424255 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.424256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.424257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.424416 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.425019 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.425066 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.425161 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":88,"threads_started":1}
I20260806 03:42:34.425848 5433 ts_manager.cc:194] Registered new tserver with Master: 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535)
I20260806 03:42:34.427069 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.409488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:39653 (local address 127.30.24.62:42619)
0806 03:42:34.409543 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.409545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.410937 (+ 1392us) server_negotiation.cc:407] Connection header received
0806 03:42:34.411058 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.411060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.411076 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.411111 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.412059 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.412964 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.414753 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.415091 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.415139 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.419882 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.419883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.419883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.419889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.421856 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.426154 (+ 4298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.426155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.426155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.426218 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.426476 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.426477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.426477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.426627 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.426672 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.426899 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.426979 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:42:34.427773 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.409197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:39653)
0806 03:42:34.410867 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.410878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.410968 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.411192 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.411193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.411682 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.411943 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.411948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.413114 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.413115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.414618 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.414621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.415251 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.415252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.415418 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.416303 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.416311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.419781 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.425805 (+ 6024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.425806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.425806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.426064 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.426297 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.426298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.426299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.426379 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.426729 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.426730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.426774 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.427356 (+ 582us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.427463 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1611,"mutex_wait_us":132,"thread_start_us":103,"threads_started":1}
I20260806 03:42:34.427920 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.411645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.30.24.61:33613)
0806 03:42:34.411872 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.411874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.414283 (+ 2409us) server_negotiation.cc:407] Connection header received
0806 03:42:34.414379 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.414380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.414394 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.414435 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.416887 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.417717 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.418727 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.419060 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.419143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.421478 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.421480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.421480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.421487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.425729 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.426967 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.426968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.426968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.427039 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.427314 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.427315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.427316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.427447 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.427531 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.427732 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.427824 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":61,"threads_started":1}
I20260806 03:42:34.428567 5888 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42619
I20260806 03:42:34.428629 5888 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.428699 5888 heartbeater.cc:511] Master 127.30.24.62:42619 requested a full tablet report, sending...
I20260806 03:42:34.428931 4645 ts_manager.cc:194] Re-registered known tserver with Master: 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535)
I20260806 03:42:34.429396 4645 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:39653
I20260806 03:42:34.427080 4646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:34.429852 4646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:34.430259 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.425151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51232)
0806 03:42:34.425238 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.425250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.425287 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.425541 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.425542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.425544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:34.425581 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.425582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.427705 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.427706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.429056 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.429059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.430016 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.430017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.430076 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.430091 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.430150 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.430199 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 03:42:34.430553 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.425353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.30.24.60:43969)
0806 03:42:34.425390 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.425392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.425399 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:42:34.425423 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.425424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.425450 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.425504 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:34.426860 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.427597 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.429292 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.429841 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.429906 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.429981 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.430414 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.430453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:42:34.430791 5443 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 03:42:34.430990 5472 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca90d7c1b9d46bb973116b7e9776a6e. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:34.431093 5472 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: This master's current role is: LEARNER
I20260806 03:42:34.431226 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: NON_VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: true } } }
I20260806 03:42:34.431326 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:34.431830 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.410839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.30.24.60:43969)
0806 03:42:34.418777 (+ 7938us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.418780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.418789 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:34.418814 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.418815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.418828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.419614 (+ 786us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.419658 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.420547 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.421637 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.421968 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.422361 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.423609 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.423611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.423611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.423617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.430007 (+ 6390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.430481 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.430482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.430482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.430771 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.431032 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.431033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.431033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.431168 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.431619 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.431661 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.431743 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7913,"thread_start_us":66,"threads_started":1}
I20260806 03:42:34.432739 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.409587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:51739 (local address 127.30.24.61:33613)
0806 03:42:34.409656 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.409658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.409672 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:42:34.410920 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.410921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.410935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.410994 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.412056 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.413006 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.414769 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.415127 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.415174 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.422959 (+ 7785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.422960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.422960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.422969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.424928 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.429160 (+ 4232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.429161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.429162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.429223 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.431735 (+ 2512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.431736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.431736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.431912 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.431970 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.432539 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.432642 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:42:34.432780 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.409197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.30.24.3:51739)
0806 03:42:34.409396 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.409444 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.410651 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.411226 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.411227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.411683 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:34.411945 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.411952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.413124 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.413125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.414661 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.414664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.415304 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.415305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.415419 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.416518 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:34.416524 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:34.420434 (+ 3910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:34.428530 (+ 8096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.428531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.428531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.429036 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.430526 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:34.430527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:34.430528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:34.430607 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:34.432333 (+ 1726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:34.432334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:34.432384 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.432507 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.432644 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":113,"thread_start_us":120,"threads_started":1}
I20260806 03:42:34.432826 5396 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: attempting to promote NON_VOTER c9ebec280f1f4fd382f5b2a482dfd5b5 to VOTER
I20260806 03:42:34.432972 5396 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:34.433476 5889 heartbeater.cc:347] Connected to a master server at 127.30.24.61:33613
I20260806 03:42:34.433496 5255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Refusing update from remote peer bca90d7c1b9d46bb973116b7e9776a6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 03:42:34.433600 5442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 LEARNER]: Refusing update from remote peer bca90d7c1b9d46bb973116b7e9776a6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 03:42:34.433661 5889 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:34.433827 5206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 03:42:34.433926 5889 heartbeater.cc:511] Master 127.30.24.61:33613 requested a full tablet report, sending...
I20260806 03:42:34.434173 5206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 03:42:34.434329 5245 ts_manager.cc:194] Registered new tserver with Master: 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535)
I20260806 03:42:34.434628 5395 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.434801 5442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.435576 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.435669 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:34.435636 5206 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.435716 5206 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:34.435751 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.435818 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:34.435864 5472 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.435854 5255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.435927 5472 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:42:34.436961 5394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:34.437057 5394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:42:34.440990 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.437039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.30.24.3:42535)
0806 03:42:34.437193 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.437198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.437493 (+ 295us) server_negotiation.cc:407] Connection header received
0806 03:42:34.437563 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.437565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.437588 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.437687 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:34.437815 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.438729 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.439825 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.440656 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.440744 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.440825 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.440880 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.440915 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260806 03:42:34.440980 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.436563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:42535 (local address 127.0.0.1:57054)
0806 03:42:34.437432 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.437442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.437498 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.437684 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.437685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.437687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:34.437737 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.437739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.438805 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.438806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.439728 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.439731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.440745 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.440746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.440805 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.440810 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.440863 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.440907 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":100,"threads_started":1}
I20260806 03:42:34.441933 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.436497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:32995 (local address 127.0.0.1:36884)
0806 03:42:34.436537 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.436549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.436595 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.438032 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.438034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.438035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:34.438069 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.438071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.439460 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.439461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.440546 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.440549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.441702 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.441704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.441765 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.441769 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.441835 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.441874 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 03:42:34.441797 5821 tablet_service.cc:1467] Processing CreateTablet for tablet 6455c25ab1834c138f4cf49a6794cca2 (DEFAULT_TABLE table=test-table [id=4d8439fdb6234f06b2afc272e934243c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:34.442030 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.436842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.30.24.2:32995)
0806 03:42:34.437860 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.437865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.437876 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:34.437916 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.437918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.437941 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.438012 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:34.438166 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.439367 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.440640 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.441618 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.441682 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.441750 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.441895 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.441943 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":99,"threads_started":1}
I20260806 03:42:34.442229 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6455c25ab1834c138f4cf49a6794cca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:34.442706 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 6455c25ab1834c138f4cf49a6794cca2 (DEFAULT_TABLE table=test-table [id=4d8439fdb6234f06b2afc272e934243c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:34.443092 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6455c25ab1834c138f4cf49a6794cca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:34.444149 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.437296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:43743 (local address 127.0.0.1:43894)
0806 03:42:34.438093 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.438102 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.438174 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.438480 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.438481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.438483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:34.438528 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.438529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.439793 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.439794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.441012 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.441015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.443872 (+ 2857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.443873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.443948 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.443953 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.444022 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.444070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":102,"threads_started":1}
I20260806 03:42:34.444355 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.437402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.30.24.1:43743)
0806 03:42:34.438309 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.438314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.438323 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:34.438364 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.438366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.438382 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.438470 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:34.438595 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.439694 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.441787 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.443326 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.443424 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.443826 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.444063 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.444111 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":83,"threads_started":1}
I20260806 03:42:34.445327 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 6455c25ab1834c138f4cf49a6794cca2 (DEFAULT_TABLE table=test-table [id=4d8439fdb6234f06b2afc272e934243c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:34.445674 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6455c25ab1834c138f4cf49a6794cca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:34.445861 5916 tablet_bootstrap.cc:492] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Bootstrap starting.
I20260806 03:42:34.446729 5916 tablet_bootstrap.cc:654] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:34.447114 5916 log.cc:824] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:34.447501 5917 tablet_bootstrap.cc:492] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Bootstrap starting.
I20260806 03:42:34.447933 5916 tablet_bootstrap.cc:492] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: No bootstrap required, opened a new log
I20260806 03:42:34.447983 5916 ts_tablet_manager.cc:1397] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:34.449127 5917 tablet_bootstrap.cc:654] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:34.449519 5917 log.cc:824] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:34.450361 5917 tablet_bootstrap.cc:492] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: No bootstrap required, opened a new log
I20260806 03:42:34.450299 5916 raft_consensus.cc:348] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.450436 5917 ts_tablet_manager.cc:1397] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 03:42:34.450470 5916 raft_consensus.cc:374] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:34.450527 5916 raft_consensus.cc:729] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b7ad462ff847b7b85ed1219d4bccce, State: Initialized, Role: FOLLOWER
I20260806 03:42:34.450901 5916 consensus_queue.cc:260] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.451213 5916 ts_tablet_manager.cc:1428] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:34.451241 5888 heartbeater.cc:503] Master 127.30.24.62:42619 was elected leader, sending a full tablet report...
I20260806 03:42:34.451834 5919 tablet_bootstrap.cc:492] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Bootstrap starting.
I20260806 03:42:34.453218 5919 tablet_bootstrap.cc:654] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:34.453651 5919 log.cc:824] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:34.454042 5917 raft_consensus.cc:348] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.454252 5917 raft_consensus.cc:374] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:34.454301 5917 raft_consensus.cc:729] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf6d272c57a4f42bd5b876560a13e2f, State: Initialized, Role: FOLLOWER
I20260806 03:42:34.454435 5917 consensus_queue.cc:260] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.454489 5919 tablet_bootstrap.cc:492] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: No bootstrap required, opened a new log
I20260806 03:42:34.454555 5919 ts_tablet_manager.cc:1397] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:42:34.454738 5917 ts_tablet_manager.cc:1428] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 03:42:34.454766 5752 heartbeater.cc:503] Master 127.30.24.62:42619 was elected leader, sending a full tablet report...
I20260806 03:42:34.456761 5919 raft_consensus.cc:348] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.456897 5919 raft_consensus.cc:374] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:34.456972 5919 raft_consensus.cc:729] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0303fb37ba5426c90b4378df948177a, State: Initialized, Role: FOLLOWER
I20260806 03:42:34.457063 5919 consensus_queue.cc:260] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.457276 5919 ts_tablet_manager.cc:1428] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 03:42:34.457361 5615 heartbeater.cc:503] Master 127.30.24.62:42619 was elected leader, sending a full tablet report...
W20260806 03:42:34.513644 5754 tablet.cc:2369] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:42:34.620023 5617 tablet.cc:2369] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:42:34.659503 5890 tablet.cc:2369] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:34.668881 5924 raft_consensus.cc:482] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:34.669018 5924 raft_consensus.cc:504] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.669392 5924 leader_election.cc:288] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995), 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535)
I20260806 03:42:34.672585 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.669732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:36031 (local address 127.30.24.2:32995)
0806 03:42:34.669798 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.669800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.669813 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:34.669926 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.669927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.669940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.670008 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:34.670183 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.670674 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.671812 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.672196 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.672280 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.672291 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.672486 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.672527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:42:34.672591 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.669592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:32995 (local address 127.30.24.1:36031)
0806 03:42:34.669756 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.669768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.669832 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.670013 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.670014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.670016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:34.670110 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.670112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.670761 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.670762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.671718 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.671728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.672283 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.672284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.672371 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.672375 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.672439 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.672503 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":22,"thread_start_us":70,"threads_started":1}
I20260806 03:42:34.672816 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.669683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:52989 (local address 127.30.24.3:42535)
0806 03:42:34.669734 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.669736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.669806 (+ 70us) server_negotiation.cc:407] Connection header received
0806 03:42:34.669909 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.669910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.669931 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.670003 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:34.670182 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.670705 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.671853 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.672412 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.672530 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.672540 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.672711 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.672753 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:34.672878 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.669576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:42535 (local address 127.30.24.1:52989)
0806 03:42:34.669728 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:34.669743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:34.669821 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:34.670008 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:34.670010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:34.670012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:34.670098 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.670100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.670810 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.670812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.671751 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:34.671754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.672537 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.672538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:34.672621 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.672624 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:34.672700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 03:42:34.672756 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260806 03:42:34.673308 5841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6455c25ab1834c138f4cf49a6794cca2" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b7ad462ff847b7b85ed1219d4bccce" is_pre_election: true
I20260806 03:42:34.673292 5705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6455c25ab1834c138f4cf49a6794cca2" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" is_pre_election: true
I20260806 03:42:34.673465 5841 raft_consensus.cc:2454] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 0.
I20260806 03:42:34.673465 5705 raft_consensus.cc:2454] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 0.
I20260806 03:42:34.673697 5501 leader_election.cc:302] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf6d272c57a4f42bd5b876560a13e2f, f0303fb37ba5426c90b4378df948177a; no voters:
I20260806 03:42:34.673878 5924 raft_consensus.cc:2781] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:34.673956 5924 raft_consensus.cc:482] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:34.673974 5924 raft_consensus.cc:3037] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.674717 5924 raft_consensus.cc:504] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.674865 5924 leader_election.cc:288] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 election: Requested vote from peers 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995), 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535)
I20260806 03:42:34.675029 5841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6455c25ab1834c138f4cf49a6794cca2" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b7ad462ff847b7b85ed1219d4bccce"
I20260806 03:42:34.675113 5841 raft_consensus.cc:3037] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.675145 5705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6455c25ab1834c138f4cf49a6794cca2" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf6d272c57a4f42bd5b876560a13e2f"
I20260806 03:42:34.675256 5705 raft_consensus.cc:3037] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.675935 5841 raft_consensus.cc:2454] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 1.
I20260806 03:42:34.676090 5705 raft_consensus.cc:2454] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 1.
I20260806 03:42:34.676128 5499 leader_election.cc:302] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38b7ad462ff847b7b85ed1219d4bccce, f0303fb37ba5426c90b4378df948177a; no voters:
I20260806 03:42:34.676239 5924 raft_consensus.cc:2781] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:34.676378 5924 raft_consensus.cc:686] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 1 LEADER]: Becoming Leader. State: Replica: f0303fb37ba5426c90b4378df948177a, State: Running, Role: LEADER
I20260806 03:42:34.676612 5924 consensus_queue.cc:237] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:34.677452 4646 catalog_manager.cc:5495] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a reported cstate change: term changed from 0 to 1, leader changed from <none> to f0303fb37ba5426c90b4378df948177a (127.30.24.1). New cstate: current_term: 1 leader_uuid: "f0303fb37ba5426c90b4378df948177a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:34.721100 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.718777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.30.24.1:43743)
0806 03:42:34.718851 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.718853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.718861 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:34.718903 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.718903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.718926 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.718998 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:34.719140 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.719649 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.720356 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.720620 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.720677 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.720828 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:34.720934 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:34.720999 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.721018 (+ 19us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.721038 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 03:42:34.723062 5841 raft_consensus.cc:1261] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Refusing update from remote peer f0303fb37ba5426c90b4378df948177a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:34.723079 5705 raft_consensus.cc:1261] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Refusing update from remote peer f0303fb37ba5426c90b4378df948177a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:34.723487 5924 consensus_queue.cc:1035] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.723675 5927 consensus_queue.cc:1035] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.727712 5931 mvcc.cc:205] Tried to move back new op lower bound from 7315405843342872576 to 7315405843156086784. Current Snapshot: MvccSnapshot[applied={T|T < 7315405843342872576}]
I20260806 03:42:34.728412 5932 mvcc.cc:205] Tried to move back new op lower bound from 7315405843342872576 to 7315405843156086784. Current Snapshot: MvccSnapshot[applied={T|T < 7315405843342872576 or (T in {7315405843342872576})}]
I20260806 03:42:34.729374 4645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:34.729475 4645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:34.729460 5930 mvcc.cc:205] Tried to move back new op lower bound from 7315405843342872576 to 7315405843156086784. Current Snapshot: MvccSnapshot[applied={T|T < 7315405843342872576 or (T in {7315405843342872576})}]
I20260806 03:42:34.731006 4645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bca90d7c1b9d46bb973116b7e9776a6e: Sending DeleteTablet for 3 replicas of tablet 6455c25ab1834c138f4cf49a6794cca2
I20260806 03:42:34.731328 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6455c25ab1834c138f4cf49a6794cca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:34 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:36884
I20260806 03:42:34.731350 5821 tablet_service.cc:1514] Processing DeleteTablet for tablet 6455c25ab1834c138f4cf49a6794cca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:34 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:57054
I20260806 03:42:34.731499 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 6455c25ab1834c138f4cf49a6794cca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:34 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:43894
I20260806 03:42:34.731724 5941 tablet_replica.cc:330] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: stopping tablet replica
I20260806 03:42:34.731837 5941 raft_consensus.cc:2229] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:34.731942 5941 raft_consensus.cc:2258] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:34.732059 5942 tablet_replica.cc:330] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: stopping tablet replica
I20260806 03:42:34.732218 5942 raft_consensus.cc:2229] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:34.732374 5942 raft_consensus.cc:2258] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:34.732792 5941 ts_tablet_manager.cc:1905] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:34.733115 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 4613
I20260806 03:42:34.734606 5942 ts_tablet_manager.cc:1905] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:34.736307 5942 ts_tablet_manager.cc:1918] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:34.736383 5942 log.cc:1196] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/6455c25ab1834c138f4cf49a6794cca2
I20260806 03:42:34.736680 5942 ts_tablet_manager.cc:1939] T 6455c25ab1834c138f4cf49a6794cca2 P f0303fb37ba5426c90b4378df948177a: Deleting consensus metadata
I20260806 03:42:34.740619 5940 tablet_replica.cc:330] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: stopping tablet replica
I20260806 03:42:34.740826 5940 raft_consensus.cc:2229] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:34.741004 5940 raft_consensus.cc:2258] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:34.741645 5940 ts_tablet_manager.cc:1905] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 03:42:34.742565 5500 connection.cc:535] server connection from 127.0.0.1:43894 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 03:42:34.743654 5940 ts_tablet_manager.cc:1918] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:34.743815 5940 log.cc:1196] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/6455c25ab1834c138f4cf49a6794cca2
I20260806 03:42:34.744201 5940 ts_tablet_manager.cc:1939] T 6455c25ab1834c138f4cf49a6794cca2 P 38b7ad462ff847b7b85ed1219d4bccce: Deleting consensus metadata
W20260806 03:42:34.744645 5774 connection.cc:441] server connection from 127.0.0.1:57054 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57054 (request call id 1) could send its response
I20260806 03:42:34.746264 5941 ts_tablet_manager.cc:1918] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:34.746430 5941 log.cc:1196] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/6455c25ab1834c138f4cf49a6794cca2
I20260806 03:42:34.746490 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.732146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.30.24.60:43969)
0806 03:42:34.732187 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.732190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.737254 (+ 5064us) server_negotiation.cc:407] Connection header received
0806 03:42:34.737356 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.737357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.737372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.737401 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.738289 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.739119 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.740314 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.740703 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.740753 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.743051 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.743053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.743053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.743063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.744975 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.745407 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.745408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.745409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.745468 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.745704 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.745705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.745705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.745851 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.745935 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.746169 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.746307 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:42:34.747059 5941 ts_tablet_manager.cc:1939] T 6455c25ab1834c138f4cf49a6794cca2 P 2bf6d272c57a4f42bd5b876560a13e2f: Deleting consensus metadata
W20260806 03:42:34.747448 5641 connection.cc:441] server connection from 127.0.0.1:36884 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36884 (request call id 1) could send its response
I20260806 03:42:34.754074 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.745652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.30.24.60:43969)
0806 03:42:34.745700 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.745701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.746511 (+ 810us) server_negotiation.cc:407] Connection header received
0806 03:42:34.746614 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.746615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.746627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.746671 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.747415 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.748022 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.748984 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.749366 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.749443 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.750948 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.750950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.750950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.750957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.752785 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.753207 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.753209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.753209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.753268 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.753499 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.753500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.753501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.753644 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.753761 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.753881 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.753958 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:42:34.755280 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:34.746021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.30.24.61:33613)
0806 03:42:34.746067 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:34.746069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:34.747510 (+ 1441us) server_negotiation.cc:407] Connection header received
0806 03:42:34.747643 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:34.747644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:34.747659 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:34.747732 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:34.748549 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.749201 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.750236 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:34.750572 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:34.750651 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:34.751922 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:34.751923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:34.751924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:34.751932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:34.753677 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.754470 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.754471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.754472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.754526 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:34.754753 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:34.754753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:34.754754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:34.754869 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:34.754972 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:34.755076 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 03:42:34.755161 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:35.727901 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:35.727573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:56691)
0806 03:42:35.727722 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:35.727826 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260806 03:42:35.728124 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:35.727762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:43971)
0806 03:42:35.727873 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:35.727958 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
W20260806 03:42:35.728204 5615 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:42619 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
W20260806 03:42:35.728592 5752 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:42619 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:35.728600 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:35.728301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:40119)
0806 03:42:35.728442 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:35.728550 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":47,"threads_started":1}
W20260806 03:42:35.728837 5888 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:42619 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:36.231700 5954 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bca90d7c1b9d46bb973116b7e9776a6e)
I20260806 03:42:36.231808 5955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bca90d7c1b9d46bb973116b7e9776a6e)
I20260806 03:42:36.231843 5954 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:36.231956 5955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:36.232224 5954 leader_election.cc:288] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619), 6f5d0dbf762d407a85a8aaa3a370025e (127.30.24.61:33613)
I20260806 03:42:36.232501 5955 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619), c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60:43969)
I20260806 03:42:36.232754 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.232384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55294)
0806 03:42:36.232642 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.232705 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260806 03:42:36.232790 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.232358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55298)
0806 03:42:36.232698 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.232748 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":48,"threads_started":1}
W20260806 03:42:36.232901 5229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
W20260806 03:42:36.232935 5416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:36.233358 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.233270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55312)
0806 03:42:36.233312 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.233334 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 03:42:36.233465 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.233357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55328)
0806 03:42:36.233403 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.233430 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 03:42:36.233469 5229 leader_election.cc:334] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
W20260806 03:42:36.233572 5416 leader_election.cc:334] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:36.236083 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.232600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:43969 (local address 127.0.0.1:51256)
0806 03:42:36.232891 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.232902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:36.232957 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:36.233143 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:36.233145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:36.233147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:36.233473 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:36.233477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.234505 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.234505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:36.235245 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:36.235247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.235814 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.235816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:36.235887 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:36.235890 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:36.235968 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 03:42:36.236015 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":51,"threads_started":1}
I20260806 03:42:36.236102 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.232708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.30.24.60:43969)
0806 03:42:36.233003 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:36.233004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:36.233012 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:36.233037 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:36.233037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:36.233048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:36.233123 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:36.233583 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.234394 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.235337 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.235726 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.235813 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:36.235826 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:36.235977 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 03:42:36.236018 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":40,"threads_started":1}
I20260806 03:42:36.236263 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.232527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.30.24.61:33613)
0806 03:42:36.232790 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:36.232791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:36.232802 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:36.232834 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:36.232835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:36.232847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:36.232929 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:36.233658 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.234393 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.235418 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.235844 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.235940 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:36.235952 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:36.236147 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 03:42:36.236196 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":53,"threads_started":1}
I20260806 03:42:36.236280 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.232393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:33613 (local address 127.0.0.1:50586)
0806 03:42:36.232653 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.232666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:36.232735 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:36.233318 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:36.233319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:36.233322 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:36.233382 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:36.233385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.234480 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.234481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:36.235310 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:36.235311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:36.235942 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:36.235943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:36.236058 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:36.236062 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:36.236141 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 03:42:36.236191 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":42,"threads_started":1}
I20260806 03:42:36.236308 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" is_pre_election: true
I20260806 03:42:36.236426 5442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f5d0dbf762d407a85a8aaa3a370025e in term 1.
I20260806 03:42:36.236547 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" is_pre_election: true
I20260806 03:42:36.236631 5231 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f5d0dbf762d407a85a8aaa3a370025e, c9ebec280f1f4fd382f5b2a482dfd5b5; no voters: bca90d7c1b9d46bb973116b7e9776a6e
I20260806 03:42:36.236680 5255 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ebec280f1f4fd382f5b2a482dfd5b5 in term 1.
I20260806 03:42:36.236745 5955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:42:36.236779 5955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bca90d7c1b9d46bb973116b7e9776a6e)
I20260806 03:42:36.236805 5955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:36.236847 5417 leader_election.cc:302] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f5d0dbf762d407a85a8aaa3a370025e, c9ebec280f1f4fd382f5b2a482dfd5b5; no voters: bca90d7c1b9d46bb973116b7e9776a6e
I20260806 03:42:36.236967 5954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:42:36.237005 5954 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bca90d7c1b9d46bb973116b7e9776a6e)
I20260806 03:42:36.237032 5954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:36.237682 5955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:36.237829 5955 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 2 election: Requested vote from peers bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619), c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60:43969)
I20260806 03:42:36.238070 5954 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:36.238147 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.237970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55334)
0806 03:42:36.238049 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.238078 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 03:42:36.238246 5954 leader_election.cc:288] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [CANDIDATE]: Term 2 election: Requested vote from peers bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619), 6f5d0dbf762d407a85a8aaa3a370025e (127.30.24.61:33613)
I20260806 03:42:36.238390 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f5d0dbf762d407a85a8aaa3a370025e"
I20260806 03:42:36.238480 5255 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c9ebec280f1f4fd382f5b2a482dfd5b5 in current term 2: Already voted for candidate 6f5d0dbf762d407a85a8aaa3a370025e in this term.
I20260806 03:42:36.238672 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.238425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55346)
0806 03:42:36.238601 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.238635 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 03:42:36.238677 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
I20260806 03:42:36.238829 5442 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f5d0dbf762d407a85a8aaa3a370025e in current term 2: Already voted for candidate c9ebec280f1f4fd382f5b2a482dfd5b5 in this term.
I20260806 03:42:36.239010 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.238908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55372)
0806 03:42:36.238960 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.238985 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 03:42:36.239084 5416 leader_election.cc:334] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:36.239135 5416 leader_election.cc:302] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9ebec280f1f4fd382f5b2a482dfd5b5; no voters: 6f5d0dbf762d407a85a8aaa3a370025e, bca90d7c1b9d46bb973116b7e9776a6e
I20260806 03:42:36.239218 5954 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:42:36.239683 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.239503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:55360)
0806 03:42:36.239604 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.239633 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 03:42:36.239745 5229 leader_election.cc:334] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:36.239804 5229 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f5d0dbf762d407a85a8aaa3a370025e; no voters: bca90d7c1b9d46bb973116b7e9776a6e, c9ebec280f1f4fd382f5b2a482dfd5b5
I20260806 03:42:36.239902 5955 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:42:36.728962 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.728659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:40295)
0806 03:42:36.728851 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.728901 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260806 03:42:36.729197 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.728979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:37103)
0806 03:42:36.729107 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.729148 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260806 03:42:36.729456 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:36.729193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:39303)
0806 03:42:36.729371 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:36.729406 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260806 03:42:37.730020 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.729763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:60621)
0806 03:42:37.729920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.729965 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260806 03:42:37.730183 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.729756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:35875)
0806 03:42:37.730087 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.730120 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":79,"threads_started":1}
I20260806 03:42:37.730225 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.729946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:57311)
0806 03:42:37.730058 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.730180 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260806 03:42:37.845112 5970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:37.845208 5970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:37.845372 5970 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619), c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60:43969)
I20260806 03:42:37.845610 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" is_pre_election: true
I20260806 03:42:37.845762 5442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f5d0dbf762d407a85a8aaa3a370025e in term 2.
I20260806 03:42:37.845728 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.845483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:44624)
0806 03:42:37.845638 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.845675 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260806 03:42:37.845943 5231 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f5d0dbf762d407a85a8aaa3a370025e, c9ebec280f1f4fd382f5b2a482dfd5b5; no voters:
I20260806 03:42:37.846065 5970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:42:37.846104 5970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:37.846127 5970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:37.846170 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.846066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:44632)
0806 03:42:37.846106 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.846145 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260806 03:42:37.846235 5229 leader_election.cc:334] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:37.846889 5970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:37.847034 5970 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 3 election: Requested vote from peers bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619), c9ebec280f1f4fd382f5b2a482dfd5b5 (127.30.24.60:43969)
I20260806 03:42:37.847226 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.847123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:44644)
0806 03:42:37.847168 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.847195 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 03:42:37.847198 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5"
I20260806 03:42:37.847286 5442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:37.847558 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.847466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:44656)
0806 03:42:37.847514 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.847538 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 03:42:37.847674 5229 leader_election.cc:334] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
I20260806 03:42:37.848085 5442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f5d0dbf762d407a85a8aaa3a370025e in term 3.
I20260806 03:42:37.848275 5231 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f5d0dbf762d407a85a8aaa3a370025e, c9ebec280f1f4fd382f5b2a482dfd5b5; no voters: bca90d7c1b9d46bb973116b7e9776a6e
I20260806 03:42:37.848366 5970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:42:37.848517 5970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [term 3 LEADER]: Becoming Leader. State: Replica: 6f5d0dbf762d407a85a8aaa3a370025e, State: Running, Role: LEADER
I20260806 03:42:37.848613 5970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } }
I20260806 03:42:37.849004 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f5d0dbf762d407a85a8aaa3a370025e. Latest consensus state: current_term: 3 leader_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:37.849107 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:37.849324 5975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:37.849799 5975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:37.849907 5975 catalog_manager.cc:1259] Loaded cluster ID: 54a42214f7124ac6939f1bfbfc3f817b
I20260806 03:42:37.849949 5975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:37.850142 5975 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:37.850243 5975 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Loaded TSK: 0
I20260806 03:42:37.850399 5975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:37.925730 5245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:37.927335 5245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:37.928503 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.928335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:44662)
0806 03:42:37.928399 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.928435 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 03:42:37.928565 5442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [term 3 FOLLOWER]: Refusing update from remote peer 6f5d0dbf762d407a85a8aaa3a370025e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 03:42:37.928843 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.928749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.0.0.1:44666)
0806 03:42:37.928789 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.928813 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 03:42:37.928830 5973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 03:42:37.928978 5229 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e -> Peer bca90d7c1b9d46bb973116b7e9776a6e (127.30.24.62:42619): Couldn't send request to peer bca90d7c1b9d46bb973116b7e9776a6e. Status: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:37.929492 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f5d0dbf762d407a85a8aaa3a370025e. Latest consensus state: current_term: 3 leader_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:37.929592 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:42:37.930120 5976 mvcc.cc:205] Tried to move back new op lower bound from 7315405856472612864 to 7315405856148766720. Current Snapshot: MvccSnapshot[applied={T|T < 7315405856472612864}]
I20260806 03:42:37.930420 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:37.930516 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ebec280f1f4fd382f5b2a482dfd5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:42:37.930655 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca90d7c1b9d46bb973116b7e9776a6e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42619 } } peers { permanent_uuid: "6f5d0dbf762d407a85a8aaa3a370025e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 33613 } attrs { promote: false } } peers { permanent_uuid: "c9ebec280f1f4fd382f5b2a482dfd5b5" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 43969 } attrs { promote: false } } }
I20260806 03:42:37.930747 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:37.935595 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.932374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.30.24.2:32995)
0806 03:42:37.932771 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:37.932774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:37.932782 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:37.932807 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:37.932808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:37.932819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:37.932893 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:37.933024 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.933688 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.934651 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.935220 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.935296 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:37.935310 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:37.935460 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 03:42:37.935519 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":55,"threads_started":1}
I20260806 03:42:37.935696 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.932160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:43743 (local address 127.0.0.1:34954)
0806 03:42:37.932309 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.932322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:37.932374 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:37.932660 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:37.932660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:37.932663 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:37.932713 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:37.932715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.933477 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.933478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:37.934772 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:37.934774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.935385 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.935386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:37.935451 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:37.935456 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:37.935508 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 03:42:37.935553 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":33,"thread_start_us":114,"threads_started":1}
I20260806 03:42:37.935758 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.932378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.30.24.1:43743)
0806 03:42:37.932528 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:37.932530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:37.932538 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:37.932569 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:37.932570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:37.932583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:37.932649 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:37.932791 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.933385 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.934877 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.935255 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.935376 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:37.935390 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:37.935619 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 03:42:37.935667 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260806 03:42:37.935853 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.932074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:42535 (local address 127.0.0.1:53850)
0806 03:42:37.932114 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.932125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:37.932174 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:37.932486 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:37.932487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:37.932489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:37.932555 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:37.932557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.933520 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.933521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:37.934772 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:37.934774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.935531 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.935532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:37.935588 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:37.935591 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:37.935624 (+ 33us) client_negotiation.cc:240] Negotiation successful
0806 03:42:37.935658 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":79}
I20260806 03:42:37.935561 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.932173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:32995 (local address 127.0.0.1:48126)
0806 03:42:37.932497 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.932507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:37.932556 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:37.932910 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:37.932911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:37.932913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:37.932947 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:37.932949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.933766 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.933767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:37.934556 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:37.934559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.935313 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.935314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:37.935383 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:37.935388 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:37.935437 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 03:42:37.935482 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":94,"threads_started":1}
I20260806 03:42:37.936004 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.932160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.30.24.3:42535)
0806 03:42:37.932295 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:37.932297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:37.932307 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:37.932338 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:37.932339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:37.932349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:37.932627 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:37.932687 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.933451 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.934860 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:37.935436 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:37.935741 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:37.935755 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:37.935775 (+ 20us) server_negotiation.cc:299] Negotiation successful
0806 03:42:37.935814 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260806 03:42:37.936034 5547 tablet_service.cc:1467] Processing CreateTablet for tablet fae897cc139e49a09d1b16e40d2b74ec (DEFAULT_TABLE table=test-table [id=73102b7d995f4ef89480b79e59250167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:37.936172 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae897cc139e49a09d1b16e40d2b74ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.936244 5821 tablet_service.cc:1467] Processing CreateTablet for tablet fae897cc139e49a09d1b16e40d2b74ec (DEFAULT_TABLE table=test-table [id=73102b7d995f4ef89480b79e59250167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:37.936376 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae897cc139e49a09d1b16e40d2b74ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.936599 5685 tablet_service.cc:1467] Processing CreateTablet for tablet fae897cc139e49a09d1b16e40d2b74ec (DEFAULT_TABLE table=test-table [id=73102b7d995f4ef89480b79e59250167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:37.936707 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae897cc139e49a09d1b16e40d2b74ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.937610 5988 tablet_bootstrap.cc:492] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a: Bootstrap starting.
I20260806 03:42:37.937879 5989 tablet_bootstrap.cc:492] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Bootstrap starting.
I20260806 03:42:37.937994 5988 tablet_bootstrap.cc:654] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:37.938227 5990 tablet_bootstrap.cc:492] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Bootstrap starting.
I20260806 03:42:37.938246 5989 tablet_bootstrap.cc:654] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:37.938575 5990 tablet_bootstrap.cc:654] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:37.938836 5989 tablet_bootstrap.cc:492] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: No bootstrap required, opened a new log
I20260806 03:42:37.938881 5989 ts_tablet_manager.cc:1397] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:37.939019 5989 raft_consensus.cc:348] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:37.939107 5989 raft_consensus.cc:374] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:37.939133 5989 raft_consensus.cc:729] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b7ad462ff847b7b85ed1219d4bccce, State: Initialized, Role: FOLLOWER
I20260806 03:42:37.939163 5990 tablet_bootstrap.cc:492] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: No bootstrap required, opened a new log
I20260806 03:42:37.939193 5989 consensus_queue.cc:260] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:37.939221 5990 ts_tablet_manager.cc:1397] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:37.939386 5989 ts_tablet_manager.cc:1428] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:37.939373 5990 raft_consensus.cc:348] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:37.939471 5990 raft_consensus.cc:374] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:37.939502 5990 raft_consensus.cc:729] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf6d272c57a4f42bd5b876560a13e2f, State: Initialized, Role: FOLLOWER
I20260806 03:42:37.939569 5990 consensus_queue.cc:260] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:37.939666 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.939563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:53025)
0806 03:42:37.939608 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.939635 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 03:42:37.939754 5990 ts_tablet_manager.cc:1428] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:37.939936 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.939854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:40113)
0806 03:42:37.939888 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.939910 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 03:42:37.940199 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.940092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:35949)
0806 03:42:37.940135 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.940170 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 03:42:37.940514 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.940395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:59609)
0806 03:42:37.940432 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.940475 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 03:42:37.940701 5988 tablet_bootstrap.cc:492] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a: No bootstrap required, opened a new log
I20260806 03:42:37.940742 5988 ts_tablet_manager.cc:1397] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 03:42:37.940824 5988 raft_consensus.cc:348] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:37.940867 5988 raft_consensus.cc:374] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:37.940881 5988 raft_consensus.cc:729] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0303fb37ba5426c90b4378df948177a, State: Initialized, Role: FOLLOWER
I20260806 03:42:37.940917 5988 consensus_queue.cc:260] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:37.941038 5988 ts_tablet_manager.cc:1428] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:37.941383 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:37.941289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:44187)
0806 03:42:37.941331 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:37.941356 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260806 03:42:38.182821 5996 raft_consensus.cc:482] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:38.182950 5996 raft_consensus.cc:504] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:38.183198 5996 leader_election.cc:288] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535), 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995)
I20260806 03:42:38.183490 5841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae897cc139e49a09d1b16e40d2b74ec" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b7ad462ff847b7b85ed1219d4bccce" is_pre_election: true
I20260806 03:42:38.183502 5705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae897cc139e49a09d1b16e40d2b74ec" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" is_pre_election: true
I20260806 03:42:38.183602 5705 raft_consensus.cc:2454] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 0.
I20260806 03:42:38.183607 5841 raft_consensus.cc:2454] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 0.
I20260806 03:42:38.183796 5501 leader_election.cc:302] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf6d272c57a4f42bd5b876560a13e2f, f0303fb37ba5426c90b4378df948177a; no voters:
I20260806 03:42:38.183903 5996 raft_consensus.cc:2781] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:38.183945 5996 raft_consensus.cc:482] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:38.183971 5996 raft_consensus.cc:3037] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:38.184789 5996 raft_consensus.cc:504] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:38.184938 5996 leader_election.cc:288] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 election: Requested vote from peers 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535), 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995)
I20260806 03:42:38.185120 5841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae897cc139e49a09d1b16e40d2b74ec" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b7ad462ff847b7b85ed1219d4bccce"
I20260806 03:42:38.185209 5841 raft_consensus.cc:3037] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:38.185153 5705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae897cc139e49a09d1b16e40d2b74ec" candidate_uuid: "f0303fb37ba5426c90b4378df948177a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf6d272c57a4f42bd5b876560a13e2f"
I20260806 03:42:38.185242 5705 raft_consensus.cc:3037] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:38.185901 5705 raft_consensus.cc:2454] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 1.
I20260806 03:42:38.185900 5841 raft_consensus.cc:2454] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0303fb37ba5426c90b4378df948177a in term 1.
I20260806 03:42:38.186151 5499 leader_election.cc:302] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38b7ad462ff847b7b85ed1219d4bccce, f0303fb37ba5426c90b4378df948177a; no voters:
I20260806 03:42:38.186272 5996 raft_consensus.cc:2781] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:38.186431 5996 raft_consensus.cc:686] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [term 1 LEADER]: Becoming Leader. State: Replica: f0303fb37ba5426c90b4378df948177a, State: Running, Role: LEADER
I20260806 03:42:38.186480 5996 consensus_queue.cc:237] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } }
I20260806 03:42:38.186690 5616 heartbeater.cc:503] Master 127.30.24.61:33613 was elected leader, sending a full tablet report...
I20260806 03:42:38.186822 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.186696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:59025)
0806 03:42:38.186747 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.186781 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 03:42:38.187398 5245 catalog_manager.cc:5495] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a reported cstate change: term changed from 0 to 1, leader changed from <none> to f0303fb37ba5426c90b4378df948177a (127.30.24.1). New cstate: current_term: 1 leader_uuid: "f0303fb37ba5426c90b4378df948177a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0303fb37ba5426c90b4378df948177a" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43743 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:38.217608 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.215277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.30.24.1:43743)
0806 03:42:38.215361 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:38.215363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:38.215371 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:38.215397 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:38.215398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:38.215410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:38.215466 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:38.215730 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:38.216229 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:38.216989 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:38.217215 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:38.217275 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:38.217405 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:38.217433 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:38.217472 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:38.217524 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 03:42:38.217547 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:42:38.218753 5841 raft_consensus.cc:1261] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Refusing update from remote peer f0303fb37ba5426c90b4378df948177a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:38.218771 5705 raft_consensus.cc:1261] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Refusing update from remote peer f0303fb37ba5426c90b4378df948177a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:38.219008 5997 consensus_queue.cc:1035] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b7ad462ff847b7b85ed1219d4bccce" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:38.219131 5996 consensus_queue.cc:1035] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf6d272c57a4f42bd5b876560a13e2f" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:38.219704 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.219588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:44695)
0806 03:42:38.219634 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.219664 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 03:42:38.219709 5753 heartbeater.cc:503] Master 127.30.24.61:33613 was elected leader, sending a full tablet report...
I20260806 03:42:38.219815 5889 heartbeater.cc:503] Master 127.30.24.61:33613 was elected leader, sending a full tablet report...
I20260806 03:42:38.220068 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.219923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:52217)
0806 03:42:38.220008 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.220034 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 03:42:38.221560 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.221428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:41715)
0806 03:42:38.221498 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.221527 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 03:42:38.221633 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.221525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.3:39741)
0806 03:42:38.221565 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.221593 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 03:42:38.222831 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.222731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.1:41597)
0806 03:42:38.222768 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.222800 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260806 03:42:38.222868 5245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:38.222949 5245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:38.224277 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.224179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42619 (local address 127.30.24.2:59681)
0806 03:42:38.224221 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.224246 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 03:42:38.224686 5245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f5d0dbf762d407a85a8aaa3a370025e: Sending DeleteTablet for 3 replicas of tablet fae897cc139e49a09d1b16e40d2b74ec
I20260806 03:42:38.224902 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet fae897cc139e49a09d1b16e40d2b74ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:38 UTC) from {username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:48126
I20260806 03:42:38.224962 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet fae897cc139e49a09d1b16e40d2b74ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:38 UTC) from {username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:34954
I20260806 03:42:38.225153 6009 tablet_replica.cc:330] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: stopping tablet replica
I20260806 03:42:38.225221 6009 raft_consensus.cc:2229] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:38.225214 5821 tablet_service.cc:1514] Processing DeleteTablet for tablet fae897cc139e49a09d1b16e40d2b74ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:38 UTC) from {username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:53850
I20260806 03:42:38.225270 6009 raft_consensus.cc:2258] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:38.225287 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet fae897cc139e49a09d1b16e40d2b74ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:38 UTC) from {username='oryx', principal='oryx/127.30.24.61@KRBTEST.COM'} at 127.0.0.1:48126
I20260806 03:42:38.225489 6010 tablet_replica.cc:330] T fae897cc139e49a09d1b16e40d2b74ec P f0303fb37ba5426c90b4378df948177a: stopping tablet replica
I20260806 03:42:38.225492 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 5392
I20260806 03:42:38.226662 6011 tablet_replica.cc:330] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: stopping tablet replica
I20260806 03:42:38.226732 6011 raft_consensus.cc:2229] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:38.226778 6011 raft_consensus.cc:2258] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:38.227094 6011 ts_tablet_manager.cc:1905] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:38.228591 6011 ts_tablet_manager.cc:1918] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:38.228700 6011 log.cc:1196] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/fae897cc139e49a09d1b16e40d2b74ec
I20260806 03:42:38.228924 6011 ts_tablet_manager.cc:1939] T fae897cc139e49a09d1b16e40d2b74ec P 38b7ad462ff847b7b85ed1219d4bccce: Deleting consensus metadata
I20260806 03:42:38.229254 5229 catalog_manager.cc:4841] TS 38b7ad462ff847b7b85ed1219d4bccce (127.30.24.3:42535): tablet fae897cc139e49a09d1b16e40d2b74ec (table test-table [id=73102b7d995f4ef89480b79e59250167]) successfully deleted
I20260806 03:42:38.230048 6009 ts_tablet_manager.cc:1905] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:38.231352 6009 ts_tablet_manager.cc:1918] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:38.231427 6009 log.cc:1196] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/fae897cc139e49a09d1b16e40d2b74ec
I20260806 03:42:38.231642 6009 ts_tablet_manager.cc:1939] T fae897cc139e49a09d1b16e40d2b74ec P 2bf6d272c57a4f42bd5b876560a13e2f: Deleting consensus metadata
I20260806 03:42:38.231976 5231 catalog_manager.cc:4841] TS 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995): tablet fae897cc139e49a09d1b16e40d2b74ec (table test-table [id=73102b7d995f4ef89480b79e59250167]) successfully deleted
W20260806 03:42:38.232154 5231 catalog_manager.cc:4808] TS 2bf6d272c57a4f42bd5b876560a13e2f (127.30.24.2:32995): delete failed for tablet fae897cc139e49a09d1b16e40d2b74ec because the tablet was not found. No further retry: Not found: Tablet not found: fae897cc139e49a09d1b16e40d2b74ec
I20260806 03:42:38.235366 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:38.235249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:43743 (local address 127.0.0.1:34972)
0806 03:42:38.235310 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:38.235337 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 03:42:38.235525 5230 catalog_manager.cc:4584] TS f0303fb37ba5426c90b4378df948177a (127.30.24.1:43743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fae897cc139e49a09d1b16e40d2b74ec: Network error: Client connection negotiation failed: client connection to 127.30.24.1:43743: connect: Connection refused (error 111)
I20260806 03:42:38.235579 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 5622
I20260806 03:42:38.243515 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 5758
I20260806 03:42:38.249323 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 5112
I20260806 03:42:38.255254 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 5299
2026-08-06T03:42:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6717 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:38 dist-test-slave-10pc krb5kdc[6014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:38 dist-test-slave-10pc krb5kdc[6014](info): set up 2 sockets
Aug 06 03:42:38 dist-test-slave-10pc krb5kdc[6014](info): commencing operation
krb5kdc: starting...
W20260806 03:42:39.310218 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab.
Entry for principal oryx/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab.
I20260806 03:42:39.439517 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38723
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab
--principal=oryx/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:39.524065 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:39.524282 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:39.524304 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:39.525557 6027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:39.525605 6027 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 03:42:39.525619 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:39.525632 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:39.525645 6027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:39.525695 6027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:39.527202 6027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.62.keytab
--principal=oryx/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:39.527451 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:39.527684 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:39.532840 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:39.533003 6027 server_base.cc:1034] running on GCE node
W20260806 03:42:39.532833 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:39.532833 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:39.533393 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:39.533722 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:39.534873 6027 hybrid_clock.cc:648] HybridClock initialized: now 1785987759534849 us; error 37 us; skew 500 ppm
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:39.537393 6027 init.cc:377] Logged in from keytab as oryx/127.30.24.62@KRBTEST.COM (short username oryx)
I20260806 03:42:39.537847 6027 webserver.cc:460] Webserver started at http://127.30.24.62:43687/ using document root <none> and password file <none>
I20260806 03:42:39.538079 6027 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:39.538147 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:39.538311 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:39.539418 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "34915d19cdf44fcd9d35e7996d9f358e"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "dc53488a90019687078d44753962e808"
server_key_iv: "6986704ce697fe00efbc9021cd46229e"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.539800 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "34915d19cdf44fcd9d35e7996d9f358e"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "dc53488a90019687078d44753962e808"
server_key_iv: "6986704ce697fe00efbc9021cd46229e"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.541272 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:39.541997 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:39.542203 6027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:39.542315 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "34915d19cdf44fcd9d35e7996d9f358e"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "dc53488a90019687078d44753962e808"
server_key_iv: "6986704ce697fe00efbc9021cd46229e"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.542397 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:39.558000 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:39.560029 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:39.560214 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:39.565354 6027 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:38723
I20260806 03:42:39.565408 6104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:38723 every 8 connection(s)
I20260806 03:42:39.565790 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:39.566483 6105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:39.568887 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Bootstrap starting.
I20260806 03:42:39.569453 6105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:39.569711 6105 log.cc:824] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:39.570415 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: No bootstrap required, opened a new log
I20260806 03:42:39.572491 6105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34915d19cdf44fcd9d35e7996d9f358e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38723 } }
I20260806 03:42:39.572654 6105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:39.572676 6105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34915d19cdf44fcd9d35e7996d9f358e, State: Initialized, Role: FOLLOWER
I20260806 03:42:39.572829 6105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34915d19cdf44fcd9d35e7996d9f358e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38723 } }
I20260806 03:42:39.572918 6105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:39.572948 6105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:39.572983 6105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:39.573490 6105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34915d19cdf44fcd9d35e7996d9f358e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38723 } }
I20260806 03:42:39.573589 6105 leader_election.cc:302] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34915d19cdf44fcd9d35e7996d9f358e; no voters:
I20260806 03:42:39.573729 6105 leader_election.cc:288] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:39.573805 6108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:39.573992 6108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [term 1 LEADER]: Becoming Leader. State: Replica: 34915d19cdf44fcd9d35e7996d9f358e, State: Running, Role: LEADER
I20260806 03:42:39.574009 6105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:39.574083 6108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34915d19cdf44fcd9d35e7996d9f358e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38723 } }
I20260806 03:42:39.574450 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34915d19cdf44fcd9d35e7996d9f358e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34915d19cdf44fcd9d35e7996d9f358e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38723 } } }
I20260806 03:42:39.574560 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:39.574452 6110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34915d19cdf44fcd9d35e7996d9f358e. Latest consensus state: current_term: 1 leader_uuid: "34915d19cdf44fcd9d35e7996d9f358e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34915d19cdf44fcd9d35e7996d9f358e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38723 } } }
I20260806 03:42:39.574672 6110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:39.574829 6116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:39.575068 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6027
I20260806 03:42:39.575208 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:39.575423 30816 external_mini_cluster.cc:1428] Setting key f67962a0ba2bbcad2da76e5f1348c222
I20260806 03:42:39.576020 6116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:39.577859 6116 catalog_manager.cc:1347] Generated new cluster ID: 4c1de5ef1588489ea4d43033dd9ed20a
I20260806 03:42:39.577909 6116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:39.587854 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.575924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.30.24.62:38723)
0806 03:42:39.576550 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:39.576560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:39.576591 (+ 31us) server_negotiation.cc:407] Connection header received
0806 03:42:39.576742 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:39.576749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:39.576819 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:39.576976 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:39.577718 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.579388 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.580416 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.581098 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.581224 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.584087 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:39.584092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:39.584092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:39.584105 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:39.586085 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.586505 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.586508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.586508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.586612 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.586835 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.586836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.586836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.587129 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:39.587279 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:39.587393 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 03:42:39.587621 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":74,"threads_started":1}
I20260806 03:42:39.594408 6116 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:39.594956 6116 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:39.598485 6116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Generated new TSK 0
I20260806 03:42:39.598654 6116 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab.
Entry for principal oryx/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab.
I20260806 03:42:39.628417 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:38723
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:39.711997 6131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:39.712250 6131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:39.712292 6131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:39.713678 6131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:39.713742 6131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:39.713786 6131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:39.713826 6131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:39.715389 6131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:38723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:39.715692 6131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:39.715929 6131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:39.721009 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:39.721043 6139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:39.721156 6131 server_base.cc:1034] running on GCE node
W20260806 03:42:39.721043 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:39.721469 6131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:39.721711 6131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:39.722864 6131 hybrid_clock.cc:648] HybridClock initialized: now 1785987759722836 us; error 42 us; skew 500 ppm
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:39.725601 6131 init.cc:377] Logged in from keytab as oryx/127.30.24.1@KRBTEST.COM (short username oryx)
I20260806 03:42:39.725982 6131 webserver.cc:460] Webserver started at http://127.30.24.1:34825/ using document root <none> and password file <none>
I20260806 03:42:39.726179 6131 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:39.726220 6131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:39.726311 6131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:39.727321 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "7356fd64d9934cd0800fdb443ca75194"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "c51e8c2fa889304f64cf80e78a82f5e7"
server_key_iv: "c9935c35d4d8a1740c6969a51578a440"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.727639 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "7356fd64d9934cd0800fdb443ca75194"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "c51e8c2fa889304f64cf80e78a82f5e7"
server_key_iv: "c9935c35d4d8a1740c6969a51578a440"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.728976 6131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 03:42:39.729722 6146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:39.729889 6131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:39.730033 6131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "7356fd64d9934cd0800fdb443ca75194"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "c51e8c2fa889304f64cf80e78a82f5e7"
server_key_iv: "c9935c35d4d8a1740c6969a51578a440"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.730116 6131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:39.745512 6131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:39.748031 6131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:39.748208 6131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:39.748831 6131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:39.748898 6131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:39.748937 6131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:39.748965 6131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:39.756151 6131 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:37873
I20260806 03:42:39.756282 6262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:37873 every 8 connection(s)
I20260806 03:42:39.756623 6131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:39.759418 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.749366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:48045 (local address 127.30.24.62:38723)
0806 03:42:39.749423 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:39.749425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:39.749628 (+ 203us) server_negotiation.cc:407] Connection header received
0806 03:42:39.750157 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:39.750159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:39.750179 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:39.750256 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:39.751171 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.751982 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.753757 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.754120 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.754193 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.756648 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:39.756649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:39.756650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:39.756657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:39.758234 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.758606 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.758607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.758607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.758662 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.758887 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.758888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.758888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.758991 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:39.759043 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:39.759282 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 03:42:39.759354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 03:42:39.759572 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.749349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.30.24.1:48045)
0806 03:42:39.749507 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:39.749544 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:39.750039 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:39.750365 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:39.750370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:39.750759 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:39.751070 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.751077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.752101 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.752102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.753637 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.753641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.754221 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.754222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.754370 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.754906 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:39.754915 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:39.756549 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:39.758310 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.758315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.758317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.758534 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.758723 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.758724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.758724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.758831 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.759047 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:39.759052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:39.759168 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 03:42:39.759253 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 03:42:39.759380 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260806 03:42:39.760067 6264 heartbeater.cc:347] Connected to a master server at 127.30.24.62:38723
I20260806 03:42:39.760154 6264 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:39.760360 6264 heartbeater.cc:511] Master 127.30.24.62:38723 requested a full tablet report, sending...
I20260806 03:42:39.760975 6058 ts_manager.cc:194] Registered new tserver with Master: 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873)
I20260806 03:42:39.761703 6058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:48045
I20260806 03:42:39.764432 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6131
I20260806 03:42:39.764590 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:39.764825 30816 external_mini_cluster.cc:1428] Setting key ef34a60582a31a654ee5aacda0a8dfcd
I20260806 03:42:39.767261 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.761236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.30.24.1:51485)
0806 03:42:39.761364 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:39.761375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:39.761435 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:39.761557 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:39.761558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:39.761736 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:39.761817 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.761819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.762494 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.762495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.763013 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.763014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.763390 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.763391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.763461 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.763920 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:39.763925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:39.764203 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:39.766372 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.766372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.766373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.766534 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.766726 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.766726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.766727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.766778 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.767031 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:39.767031 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:39.767066 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 03:42:39.767114 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 03:42:39.767197 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260806 03:42:39.767488 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.761309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:51485 (local address 127.30.24.62:38723)
0806 03:42:39.761358 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:39.761359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:39.761419 (+ 60us) server_negotiation.cc:407] Connection header received
0806 03:42:39.761492 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:39.761492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:39.761502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:39.761552 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:39.761880 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.762392 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.763102 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.763315 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.763390 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.764532 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:39.764534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:39.764534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:39.764542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:39.766283 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.766602 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.766603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.766604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.766665 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.766838 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.766839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.766839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.766971 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:39.767038 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:39.767140 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 03:42:39.767224 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
WARNING: no policy specified for oryx/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab.
Entry for principal oryx/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab.
I20260806 03:42:39.793021 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:38723
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:39.883981 6273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:39.884232 6273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:39.884276 6273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:39.885540 6273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:39.885609 6273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:39.885646 6273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:39.885681 6273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:39.887269 6273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:38723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:39.887557 6273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:39.887806 6273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:39.892948 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:39.892938 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:39.893103 6273 server_base.cc:1034] running on GCE node
W20260806 03:42:39.892971 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:39.893510 6273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:39.893777 6273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:39.894946 6273 hybrid_clock.cc:648] HybridClock initialized: now 1785987759894928 us; error 35 us; skew 500 ppm
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:39.897653 6273 init.cc:377] Logged in from keytab as oryx/127.30.24.2@KRBTEST.COM (short username oryx)
I20260806 03:42:39.898114 6273 webserver.cc:460] Webserver started at http://127.30.24.2:37041/ using document root <none> and password file <none>
I20260806 03:42:39.898403 6273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:39.898452 6273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:39.898602 6273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:39.899683 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "ad6738b639624d09a3323a2c95ae6c2b"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "6055edb8c4667aaa0008fe6fd61ccfe6"
server_key_iv: "7412a4ac468a090ec96985cc2603b61c"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.900051 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "ad6738b639624d09a3323a2c95ae6c2b"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "6055edb8c4667aaa0008fe6fd61ccfe6"
server_key_iv: "7412a4ac468a090ec96985cc2603b61c"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.901496 6273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:39.902334 6288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:39.902567 6273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:39.902642 6273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "ad6738b639624d09a3323a2c95ae6c2b"
format_stamp: "Formatted at 2026-08-06 03:42:39 on dist-test-slave-10pc"
server_key: "6055edb8c4667aaa0008fe6fd61ccfe6"
server_key_iv: "7412a4ac468a090ec96985cc2603b61c"
server_key_version: "encryptionkey@0"
I20260806 03:42:39.902740 6273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:39.917563 6273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:39.919544 6273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:39.919710 6273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:39.920238 6273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:39.920279 6273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:39.920303 6273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:39.920341 6273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:39.927717 6273 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:44431
I20260806 03:42:39.927806 6404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:44431 every 8 connection(s)
I20260806 03:42:39.928103 6273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:42:39 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:39.928601 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6273
I20260806 03:42:39.928709 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:39.928956 30816 external_mini_cluster.cc:1428] Setting key 4a7fc792ee4c50802a22d445fc36e5cc
I20260806 03:42:39.931891 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.920783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:38645 (local address 127.30.24.62:38723)
0806 03:42:39.920859 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:39.920861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:39.921114 (+ 253us) server_negotiation.cc:407] Connection header received
0806 03:42:39.921904 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:39.921905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:39.921916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:39.921990 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:39.922951 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.923812 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.925538 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.925825 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.925886 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.928765 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:39.928767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:39.928767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:39.928777 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:39.930296 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.930808 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.930808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.930809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.930863 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.931148 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.931149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.931150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.931268 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:39.931346 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:39.931674 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 03:42:39.931768 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 03:42:39.931923 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.920761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.30.24.2:38645)
0806 03:42:39.920961 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:39.921011 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:39.921775 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:39.922078 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:39.922083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:39.922504 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:39.922839 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.922847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.923920 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.923921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.925441 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.925444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.925919 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.925920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.926097 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.926721 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:39.926733 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:39.928643 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:39.930384 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.930398 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.930400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.930729 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.930930 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.930931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.930931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.931080 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.931335 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:39.931341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:39.931514 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 03:42:39.931633 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 03:42:39.931767 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260806 03:42:39.932374 6405 heartbeater.cc:347] Connected to a master server at 127.30.24.62:38723
I20260806 03:42:39.932487 6405 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:39.932667 6405 heartbeater.cc:511] Master 127.30.24.62:38723 requested a full tablet report, sending...
I20260806 03:42:39.933146 6059 ts_manager.cc:194] Registered new tserver with Master: ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431)
I20260806 03:42:39.933701 6059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:38645
WARNING: no policy specified for oryx/127.30.24.3@KRBTEST.COM; defaulting to no policy
I20260806 03:42:39.940245 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.934016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.30.24.2:56493)
0806 03:42:39.934193 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:39.934202 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:39.934253 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:39.934381 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:39.934382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:39.934536 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:39.934586 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.934587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.935522 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.935523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.936109 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:39.936111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.936664 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.936665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:39.936716 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.937191 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:39.937203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:39.937477 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:39.939186 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.939187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.939187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.939353 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.939573 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:39.939573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:39.939573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:39.939633 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:39.939985 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:39.939985 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:39.940021 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 03:42:39.940071 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:39.940154 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260806 03:42:39.940304 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:39.934078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56493 (local address 127.30.24.62:38723)
0806 03:42:39.934120 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:39.934122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:39.934267 (+ 145us) server_negotiation.cc:407] Connection header received
0806 03:42:39.934294 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:39.934295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:39.934306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:39.934371 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:39.934663 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.935427 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.936206 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:39.936575 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:39.936656 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:39.937565 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:39.937567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:39.937567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:39.937574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:39.939102 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.939431 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.939432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.939432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.939497 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:39.939702 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:39.939704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:39.939704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:39.939912 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:39.939991 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:39.940094 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 03:42:39.940173 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab.
Entry for principal oryx/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab.
I20260806 03:42:39.959564 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:38723
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:40.044425 6414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:40.044734 6414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:40.044764 6414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:40.046073 6414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:40.046119 6414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:40.046141 6414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 03:42:40.046182 6414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:40.047729 6414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.24.62:38723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:40.047991 6414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:40.048240 6414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:40.053562 6419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:40.053562 6420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:40.053609 6422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:40.053835 6414 server_base.cc:1034] running on GCE node
I20260806 03:42:40.054095 6414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:40.054380 6414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:40.055555 6414 hybrid_clock.cc:648] HybridClock initialized: now 1785987760055542 us; error 42 us; skew 500 ppm
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987760, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:40.058228 6414 init.cc:377] Logged in from keytab as oryx/127.30.24.3@KRBTEST.COM (short username oryx)
I20260806 03:42:40.058689 6414 webserver.cc:460] Webserver started at http://127.30.24.3:40387/ using document root <none> and password file <none>
I20260806 03:42:40.058925 6414 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:40.058995 6414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:40.059156 6414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:40.060256 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
format_stamp: "Formatted at 2026-08-06 03:42:40 on dist-test-slave-10pc"
server_key: "d8a1e36c6afed1e28e1b67a0b1115f40"
server_key_iv: "e2f2cd3705f41b7064c1f375b4a50320"
server_key_version: "encryptionkey@0"
I20260806 03:42:40.060724 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
format_stamp: "Formatted at 2026-08-06 03:42:40 on dist-test-slave-10pc"
server_key: "d8a1e36c6afed1e28e1b67a0b1115f40"
server_key_iv: "e2f2cd3705f41b7064c1f375b4a50320"
server_key_version: "encryptionkey@0"
I20260806 03:42:40.062096 6414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.062911 6429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.063112 6414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.063247 6414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
format_stamp: "Formatted at 2026-08-06 03:42:40 on dist-test-slave-10pc"
server_key: "d8a1e36c6afed1e28e1b67a0b1115f40"
server_key_iv: "e2f2cd3705f41b7064c1f375b4a50320"
server_key_version: "encryptionkey@0"
I20260806 03:42:40.063324 6414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:40.091652 6414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:40.093856 6414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:40.094038 6414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:40.094614 6414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:40.094686 6414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.094736 6414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:40.094764 6414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987760, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:40.102373 6414 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37089
I20260806 03:42:40.102831 6414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:40.103511 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37089 every 8 connection(s)
I20260806 03:42:40.105774 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.095203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:47253 (local address 127.30.24.62:38723)
0806 03:42:40.095274 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.095277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.095456 (+ 179us) server_negotiation.cc:407] Connection header received
0806 03:42:40.096028 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.096029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.096044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.096137 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.097203 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.097796 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.099511 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.099739 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.099830 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.102914 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:40.102916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:40.102916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:40.102925 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:40.104514 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.104909 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.104910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.104910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.104964 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.105153 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.105153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.105154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.105262 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:40.105312 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.105627 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.105710 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 03:42:40.105901 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6414
I20260806 03:42:40.105993 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:42:40.106073 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.095141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.30.24.3:47253)
0806 03:42:40.095335 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.095374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.095912 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.096258 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.096264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.096727 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:40.097063 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.097070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.097908 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.097909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.099402 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.099406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.099859 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.099861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.100001 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.100721 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:40.100732 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:40.102798 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:40.104608 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:40.104615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:40.104617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:40.104843 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:40.105018 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:40.105018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:40.105018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:40.105103 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:40.105322 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:40.105328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:40.105449 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.105603 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.105824 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260806 03:42:40.106220 30816 external_mini_cluster.cc:1428] Setting key f28bc94640d4fbc8a4314d8a9b3b756a
I20260806 03:42:40.106639 6547 heartbeater.cc:347] Connected to a master server at 127.30.24.62:38723
I20260806 03:42:40.106763 6547 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:40.107018 6547 heartbeater.cc:511] Master 127.30.24.62:38723 requested a full tablet report, sending...
I20260806 03:42:40.107376 6059 ts_manager.cc:194] Registered new tserver with Master: 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.107954 6059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:47253
I20260806 03:42:40.113205 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.107778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:48731 (local address 127.30.24.62:38723)
0806 03:42:40.107817 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.107819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.107887 (+ 68us) server_negotiation.cc:407] Connection header received
0806 03:42:40.107956 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.107957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.107965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.108028 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.108396 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.109050 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.109739 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.109912 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.109963 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.110842 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:40.110843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:40.110844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:40.110850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:40.112209 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.112598 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.112599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.112599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.112651 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.112828 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.112829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.112829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.112914 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:40.112963 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.113089 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.113143 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:42:40.113265 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.107707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.30.24.3:48731)
0806 03:42:40.107839 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.107849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.107901 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.108018 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.108019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.108231 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:40.108306 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.108308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.109162 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.109163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.109667 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.109668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.110038 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.110039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.110103 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.110483 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:40.110487 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:40.110763 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:40.112289 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:40.112290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:40.112290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:40.112520 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:40.112718 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:40.112718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:40.112719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:40.112776 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:40.112957 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:40.112957 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:40.112990 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.113075 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.113167 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260806 03:42:40.118178 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 03:42:40.141253 6126 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:40.135592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.30.24.62:38723)
0806 03:42:40.135659 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.135661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.135670 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:40.135723 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.135724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.135733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.135791 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.136359 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.136917 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.137621 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.137800 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.137849 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.141145 (+ 3296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51524: BlockingRecv error: recv got EOF from 127.0.0.1:51524 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:40.148861 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.142055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.30.24.62:38723)
0806 03:42:40.142118 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.142120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.142220 (+ 100us) server_negotiation.cc:407] Connection header received
0806 03:42:40.142267 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.142268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.142277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.142326 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.142841 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.143308 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.143979 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.144149 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.144207 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.146566 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:40.146567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:40.146567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:40.146574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:40.147804 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.148112 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.148112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.148113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.148166 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.148332 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.148333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.148333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.148436 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:40.148617 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.148726 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.148791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 03:42:40.150570 6059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:40.152225 6059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:40.158890 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.155142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.30.24.2:44431)
0806 03:42:40.155386 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.155391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.155401 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:40.155446 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.155448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.155463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.155539 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.155750 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.156645 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.157884 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.158400 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.158504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.158587 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.158750 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.158793 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":118,"threads_started":1}
I20260806 03:42:40.158880 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.154732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:44431 (local address 127.0.0.1:49962)
0806 03:42:40.155098 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.155109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.155166 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.155613 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.155614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.155616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.155661 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.155662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.156725 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.156726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.157794 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.157797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.158508 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.158509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.158606 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.158620 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.158729 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.158785 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":12,"thread_start_us":98,"threads_started":1}
I20260806 03:42:40.159475 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.154730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37089 (local address 127.0.0.1:42468)
0806 03:42:40.154859 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.154902 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.154965 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.155217 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.155221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.155224 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.155300 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.155302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.156665 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.156666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.157844 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.157847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.158855 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.158856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.158917 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.158921 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.159303 (+ 382us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.159361 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":38,"thread_start_us":77,"threads_started":1}
I20260806 03:42:40.159761 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.154956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37873 (local address 127.0.0.1:33228)
0806 03:42:40.155695 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.155705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.155759 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.155964 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.155965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.155967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.156005 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.156006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.157335 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.157336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.158424 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.158426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.159355 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.159356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.159419 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.159424 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.159525 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.159680 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":83,"threads_started":1}
I20260806 03:42:40.159732 6338 tablet_service.cc:1467] Processing CreateTablet for tablet a914c2259e59434ea98368313bede9f1 (DEFAULT_TABLE table=test-table [id=5457181b25e647e497daf5c6fa1612d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.160003 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.154830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.30.24.3:37089)
0806 03:42:40.154989 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.154993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.155007 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:42:40.155076 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.155078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.155096 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.155194 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.155404 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.156580 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.157945 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.158685 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.158749 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.158821 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.159856 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.159900 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260806 03:42:40.160151 6338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c2259e59434ea98368313bede9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.160574 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.155459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.30.24.1:37873)
0806 03:42:40.155641 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.155645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.155813 (+ 168us) server_negotiation.cc:407] Connection header received
0806 03:42:40.155855 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.155858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.155873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.155944 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.156089 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.157172 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.158520 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.159273 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.159342 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.159414 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.159735 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.159784 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260806 03:42:40.160735 6480 tablet_service.cc:1467] Processing CreateTablet for tablet a914c2259e59434ea98368313bede9f1 (DEFAULT_TABLE table=test-table [id=5457181b25e647e497daf5c6fa1612d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.161083 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c2259e59434ea98368313bede9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.161393 6197 tablet_service.cc:1467] Processing CreateTablet for tablet a914c2259e59434ea98368313bede9f1 (DEFAULT_TABLE table=test-table [id=5457181b25e647e497daf5c6fa1612d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.161883 6197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914c2259e59434ea98368313bede9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.163957 6575 tablet_bootstrap.cc:492] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Bootstrap starting.
I20260806 03:42:40.164664 6575 tablet_bootstrap.cc:654] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.165210 6575 log.cc:824] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:40.165293 6576 tablet_bootstrap.cc:492] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Bootstrap starting.
I20260806 03:42:40.165922 6576 tablet_bootstrap.cc:654] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.166102 6575 tablet_bootstrap.cc:492] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: No bootstrap required, opened a new log
I20260806 03:42:40.166190 6575 ts_tablet_manager.cc:1397] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:42:40.166268 6578 tablet_bootstrap.cc:492] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Bootstrap starting.
I20260806 03:42:40.166294 6576 log.cc:824] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:40.167038 6578 tablet_bootstrap.cc:654] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.167198 6576 tablet_bootstrap.cc:492] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: No bootstrap required, opened a new log
I20260806 03:42:40.167273 6576 ts_tablet_manager.cc:1397] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:40.167393 6578 log.cc:824] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:40.168128 6578 tablet_bootstrap.cc:492] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: No bootstrap required, opened a new log
I20260806 03:42:40.168198 6578 ts_tablet_manager.cc:1397] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:42:40.168305 6575 raft_consensus.cc:348] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.168525 6575 raft_consensus.cc:374] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.168576 6575 raft_consensus.cc:729] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6738b639624d09a3323a2c95ae6c2b, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.168694 6575 consensus_queue.cc:260] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.168987 6575 ts_tablet_manager.cc:1428] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:40.169014 6405 heartbeater.cc:503] Master 127.30.24.62:38723 was elected leader, sending a full tablet report...
I20260806 03:42:40.169488 6576 raft_consensus.cc:348] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.169646 6576 raft_consensus.cc:374] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.169690 6576 raft_consensus.cc:729] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5720a0d2b86242d4b4cfc64c910d0fbf, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.169816 6576 consensus_queue.cc:260] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.170137 6576 ts_tablet_manager.cc:1428] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:40.170145 6547 heartbeater.cc:503] Master 127.30.24.62:38723 was elected leader, sending a full tablet report...
I20260806 03:42:40.170346 6578 raft_consensus.cc:348] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.170534 6578 raft_consensus.cc:374] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.170581 6578 raft_consensus.cc:729] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.170694 6578 consensus_queue.cc:260] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.171005 6578 ts_tablet_manager.cc:1428] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 03:42:40.171038 6264 heartbeater.cc:503] Master 127.30.24.62:38723 was elected leader, sending a full tablet report...
W20260806 03:42:40.179543 6406 tablet.cc:2369] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:42:40.257611 6265 tablet.cc:2369] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:40.259554 6583 raft_consensus.cc:482] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:40.259675 6583 raft_consensus.cc:504] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.260100 6583 leader_election.cc:288] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.263167 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.260419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:53323 (local address 127.30.24.3:37089)
0806 03:42:40.260501 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.260503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.260607 (+ 104us) server_negotiation.cc:407] Connection header received
0806 03:42:40.260679 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.260680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.260694 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.260785 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.260954 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.261424 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.262447 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.262811 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.262893 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.262904 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.263060 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.263096 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:42:40.263173 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.260381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:35675 (local address 127.30.24.2:44431)
0806 03:42:40.260474 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.260476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.260535 (+ 59us) server_negotiation.cc:407] Connection header received
0806 03:42:40.260628 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.260629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.260643 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.260727 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.260903 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.261464 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.262440 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.262816 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.262903 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.262913 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.263064 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.263101 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:42:40.263197 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.260303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:44431 (local address 127.30.24.1:35675)
0806 03:42:40.260447 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.260487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.260566 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.260740 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.260741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.260744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.260827 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.260829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.261551 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.261552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.262342 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.262344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.262908 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.262909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.262982 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.262991 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.263058 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.263100 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260806 03:42:40.263202 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.260327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37089 (local address 127.30.24.1:53323)
0806 03:42:40.260505 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.260515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.260581 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.260805 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.260806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.260808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.260883 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.260884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.261517 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.261518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.262338 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.262341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.262894 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.262895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.262975 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.262979 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.263041 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.263094 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":15,"thread_start_us":61,"threads_started":1}
I20260806 03:42:40.263615 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c2259e59434ea98368313bede9f1" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b" is_pre_election: true
I20260806 03:42:40.263808 6359 raft_consensus.cc:2454] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.263787 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c2259e59434ea98368313bede9f1" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" is_pre_election: true
I20260806 03:42:40.263918 6501 raft_consensus.cc:2454] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.264050 6150 leader_election.cc:302] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7356fd64d9934cd0800fdb443ca75194, ad6738b639624d09a3323a2c95ae6c2b; no voters:
I20260806 03:42:40.264202 6583 raft_consensus.cc:2781] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:40.264261 6583 raft_consensus.cc:482] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:40.264284 6583 raft_consensus.cc:3037] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.265091 6583 raft_consensus.cc:504] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.265305 6583 leader_election.cc:288] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Requested vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.265398 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c2259e59434ea98368313bede9f1" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
I20260806 03:42:40.265481 6501 raft_consensus.cc:3037] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.266225 6501 raft_consensus.cc:2454] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.266359 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914c2259e59434ea98368313bede9f1" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b"
I20260806 03:42:40.266405 6148 leader_election.cc:302] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, 7356fd64d9934cd0800fdb443ca75194; no voters:
I20260806 03:42:40.266444 6359 raft_consensus.cc:3037] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.266489 6583 raft_consensus.cc:2781] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:40.266657 6583 raft_consensus.cc:686] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Becoming Leader. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Running, Role: LEADER
I20260806 03:42:40.266777 6583 consensus_queue.cc:237] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.267392 6359 raft_consensus.cc:2454] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.267817 6059 catalog_manager.cc:5495] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:40.276744 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.274254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.30.24.1:37873)
0806 03:42:40.274321 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.274323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.274332 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:40.274374 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.274375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.274394 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.274465 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.274598 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.275205 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.276018 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.276205 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.276263 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.276404 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.276569 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.276637 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.276670 (+ 33us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.276689 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:42:40.278802 6359 raft_consensus.cc:1261] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.278807 6501 raft_consensus.cc:1261] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.279191 6586 consensus_queue.cc:1035] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.279336 6583 consensus_queue.cc:1035] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.282111 6591 mvcc.cc:205] Tried to move back new op lower bound from 7315405866098724864 to 7315405866053406720. Current Snapshot: MvccSnapshot[applied={T|T < 7315405866098724864}]
I20260806 03:42:40.282889 6590 mvcc.cc:205] Tried to move back new op lower bound from 7315405866098724864 to 7315405866053406720. Current Snapshot: MvccSnapshot[applied={T|T < 7315405866098724864 or (T in {7315405866098724864})}]
I20260806 03:42:40.284085 6059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.284170 6059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.284914 6059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Sending DeleteTablet for 3 replicas of tablet a914c2259e59434ea98368313bede9f1
I20260806 03:42:40.285280 6338 tablet_service.cc:1514] Processing DeleteTablet for tablet a914c2259e59434ea98368313bede9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:49962
I20260806 03:42:40.285312 6197 tablet_service.cc:1514] Processing DeleteTablet for tablet a914c2259e59434ea98368313bede9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33228
I20260806 03:42:40.285514 6480 tablet_service.cc:1514] Processing DeleteTablet for tablet a914c2259e59434ea98368313bede9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:42468
I20260806 03:42:40.285575 6598 tablet_replica.cc:330] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: stopping tablet replica
I20260806 03:42:40.285749 6600 tablet_replica.cc:330] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: stopping tablet replica
I20260806 03:42:40.285768 6059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:40.285938 6059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:40.285960 6599 tablet_replica.cc:330] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: stopping tablet replica
I20260806 03:42:40.286053 6599 raft_consensus.cc:2229] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.286146 6600 raft_consensus.cc:2229] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.286186 6599 raft_consensus.cc:2258] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.286578 6600 raft_consensus.cc:2258] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.286685 6599 ts_tablet_manager.cc:1905] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.287133 6600 ts_tablet_manager.cc:1905] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.287215 6197 tablet_service.cc:1467] Processing CreateTablet for tablet cdc0939591d440f8a65369492e6b26ed (DEFAULT_TABLE table=test-table [id=f42d026aaa6545999afdf37a27cd54bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.287350 6197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc0939591d440f8a65369492e6b26ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.288616 6599 ts_tablet_manager.cc:1918] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:40.288695 6599 log.cc:1196] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/a914c2259e59434ea98368313bede9f1
I20260806 03:42:40.288985 6578 tablet_bootstrap.cc:492] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Bootstrap starting.
I20260806 03:42:40.289305 6600 ts_tablet_manager.cc:1918] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:40.289402 6600 log.cc:1196] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/a914c2259e59434ea98368313bede9f1
I20260806 03:42:40.289480 6578 tablet_bootstrap.cc:654] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.289707 6600 ts_tablet_manager.cc:1939] T a914c2259e59434ea98368313bede9f1 P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting consensus metadata
I20260806 03:42:40.288902 6338 tablet_service.cc:1467] Processing CreateTablet for tablet cdc0939591d440f8a65369492e6b26ed (DEFAULT_TABLE table=test-table [id=f42d026aaa6545999afdf37a27cd54bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.288985 6599 ts_tablet_manager.cc:1939] T a914c2259e59434ea98368313bede9f1 P ad6738b639624d09a3323a2c95ae6c2b: Deleting consensus metadata
I20260806 03:42:40.290184 6044 catalog_manager.cc:4841] TS 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089): tablet a914c2259e59434ea98368313bede9f1 (table test-table [id=5457181b25e647e497daf5c6fa1612d4]) successfully deleted
I20260806 03:42:40.290429 6480 tablet_service.cc:1467] Processing CreateTablet for tablet cdc0939591d440f8a65369492e6b26ed (DEFAULT_TABLE table=test-table [id=f42d026aaa6545999afdf37a27cd54bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.290540 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc0939591d440f8a65369492e6b26ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.290524 6046 catalog_manager.cc:4841] TS ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431): tablet a914c2259e59434ea98368313bede9f1 (table test-table [id=5457181b25e647e497daf5c6fa1612d4]) successfully deleted
I20260806 03:42:40.291428 6576 tablet_bootstrap.cc:492] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Bootstrap starting.
I20260806 03:42:40.290017 6338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc0939591d440f8a65369492e6b26ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.291846 6576 tablet_bootstrap.cc:654] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.292519 6576 tablet_bootstrap.cc:492] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: No bootstrap required, opened a new log
I20260806 03:42:40.292572 6576 ts_tablet_manager.cc:1397] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.292670 6575 tablet_bootstrap.cc:492] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Bootstrap starting.
I20260806 03:42:40.292743 6576 raft_consensus.cc:348] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.292826 6576 raft_consensus.cc:374] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.292855 6576 raft_consensus.cc:729] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5720a0d2b86242d4b4cfc64c910d0fbf, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.292898 6576 consensus_queue.cc:260] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.292989 6576 ts_tablet_manager.cc:1428] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.293118 6575 tablet_bootstrap.cc:654] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.293854 6575 tablet_bootstrap.cc:492] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: No bootstrap required, opened a new log
I20260806 03:42:40.293905 6575 ts_tablet_manager.cc:1397] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.294073 6575 raft_consensus.cc:348] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.294152 6575 raft_consensus.cc:374] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.294180 6575 raft_consensus.cc:729] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6738b639624d09a3323a2c95ae6c2b, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.294229 6575 consensus_queue.cc:260] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.294322 6575 ts_tablet_manager.cc:1428] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.294471 6598 raft_consensus.cc:2229] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:40.294631 6598 raft_consensus.cc:2258] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.295295 6578 tablet_bootstrap.cc:492] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: No bootstrap required, opened a new log
I20260806 03:42:40.295346 6578 ts_tablet_manager.cc:1397] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260806 03:42:40.295498 6578 raft_consensus.cc:348] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.295554 6578 raft_consensus.cc:374] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.295598 6578 raft_consensus.cc:729] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.295650 6578 consensus_queue.cc:260] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.295742 6578 ts_tablet_manager.cc:1428] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.296054 6598 ts_tablet_manager.cc:1905] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.297520 6598 ts_tablet_manager.cc:1918] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:40.297757 6598 log.cc:1196] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/a914c2259e59434ea98368313bede9f1
I20260806 03:42:40.298028 6598 ts_tablet_manager.cc:1939] T a914c2259e59434ea98368313bede9f1 P 7356fd64d9934cd0800fdb443ca75194: Deleting consensus metadata
I20260806 03:42:40.298447 6046 catalog_manager.cc:4841] TS 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873): tablet a914c2259e59434ea98368313bede9f1 (table test-table [id=5457181b25e647e497daf5c6fa1612d4]) successfully deleted
I20260806 03:42:40.308712 6586 raft_consensus.cc:482] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:40.308804 6586 raft_consensus.cc:504] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.308960 6586 leader_election.cc:288] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.309163 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc0939591d440f8a65369492e6b26ed" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" is_pre_election: true
I20260806 03:42:40.309297 6501 raft_consensus.cc:2454] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.309327 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc0939591d440f8a65369492e6b26ed" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b" is_pre_election: true
I20260806 03:42:40.309427 6359 raft_consensus.cc:2454] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.309479 6148 leader_election.cc:302] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, 7356fd64d9934cd0800fdb443ca75194; no voters:
I20260806 03:42:40.309574 6586 raft_consensus.cc:2781] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:40.309602 6586 raft_consensus.cc:482] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:40.309623 6586 raft_consensus.cc:3037] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.310518 6586 raft_consensus.cc:504] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.310657 6586 leader_election.cc:288] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Requested vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.310825 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc0939591d440f8a65369492e6b26ed" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b"
I20260806 03:42:40.310904 6359 raft_consensus.cc:3037] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.310840 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc0939591d440f8a65369492e6b26ed" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
I20260806 03:42:40.310940 6501 raft_consensus.cc:3037] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.311662 6501 raft_consensus.cc:2454] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.311689 6359 raft_consensus.cc:2454] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.311839 6148 leader_election.cc:302] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, 7356fd64d9934cd0800fdb443ca75194; no voters:
I20260806 03:42:40.311973 6586 raft_consensus.cc:2781] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:40.312052 6586 raft_consensus.cc:686] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Becoming Leader. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Running, Role: LEADER
I20260806 03:42:40.312096 6586 consensus_queue.cc:237] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.312690 6059 catalog_manager.cc:5495] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } health_report { overall_health: UNKNOWN } } }
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.62@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:40.344956 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.337378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.30.24.62:38723)
0806 03:42:40.337435 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.337438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.337544 (+ 106us) server_negotiation.cc:407] Connection header received
0806 03:42:40.337660 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.337661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.337671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.337742 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.338227 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.338726 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.339453 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.339639 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.339740 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.342201 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:40.342203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:40.342203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:40.342227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:40.343673 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.344066 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.344067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.344067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.344136 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.344346 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.344347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.344347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.344521 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:40.344586 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.344753 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.344841 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 03:42:40.345042 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.337262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38723 (local address 127.0.0.1:51554)
0806 03:42:40.337447 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.337459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.337566 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.337747 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.337748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.338079 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:40.338132 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.338135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.338810 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.338811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.339360 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.339362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.339740 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.339741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.339795 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.340401 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:40.340406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:40.342093 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:40.343776 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:40.343785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:40.343787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:40.343987 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:40.344206 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:40.344207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:40.344207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:40.344281 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:40.344576 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:40.344584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:40.344639 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.344728 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.344897 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":66,"thread_start_us":98,"threads_started":1}
I20260806 03:42:40.347671 6059 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:51554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 03:42:40.347892 6059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:40.349370 6197 tablet_service.cc:1467] Processing CreateTablet for tablet a208125ba1784a869b9f0d898da10f53 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40a9d38188c042309d9e62ae7897bede]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:40.349396 6480 tablet_service.cc:1467] Processing CreateTablet for tablet a208125ba1784a869b9f0d898da10f53 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40a9d38188c042309d9e62ae7897bede]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:40.349449 6338 tablet_service.cc:1467] Processing CreateTablet for tablet a208125ba1784a869b9f0d898da10f53 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40a9d38188c042309d9e62ae7897bede]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 03:42:40.349538 6197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a208125ba1784a869b9f0d898da10f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.349579 6338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a208125ba1784a869b9f0d898da10f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.349578 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a208125ba1784a869b9f0d898da10f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.351141 6575 tablet_bootstrap.cc:492] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: Bootstrap starting.
I20260806 03:42:40.351176 6576 tablet_bootstrap.cc:492] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: Bootstrap starting.
I20260806 03:42:40.351544 6575 tablet_bootstrap.cc:654] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.351634 6576 tablet_bootstrap.cc:654] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.352001 6578 tablet_bootstrap.cc:492] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: Bootstrap starting.
I20260806 03:42:40.352250 6575 tablet_bootstrap.cc:492] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: No bootstrap required, opened a new log
I20260806 03:42:40.352286 6575 ts_tablet_manager.cc:1397] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.352284 6578 tablet_bootstrap.cc:654] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.352413 6575 raft_consensus.cc:348] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.352528 6575 raft_consensus.cc:374] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.352551 6575 raft_consensus.cc:729] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6738b639624d09a3323a2c95ae6c2b, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.352599 6575 consensus_queue.cc:260] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.352701 6575 ts_tablet_manager.cc:1428] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:40.352912 6581 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.353012 6581 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:40.353097 6578 tablet_bootstrap.cc:492] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: No bootstrap required, opened a new log
I20260806 03:42:40.353152 6578 ts_tablet_manager.cc:1397] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:40.353302 6578 raft_consensus.cc:348] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.353379 6578 raft_consensus.cc:374] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.353401 6578 raft_consensus.cc:729] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.353449 6578 consensus_queue.cc:260] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.353574 6578 ts_tablet_manager.cc:1428] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.353614 6586 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.353713 6586 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 03:42:40.353880 6548 tablet.cc:2369] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:40.354333 6576 tablet_bootstrap.cc:492] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: No bootstrap required, opened a new log
I20260806 03:42:40.354378 6576 ts_tablet_manager.cc:1397] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 03:42:40.354501 6576 raft_consensus.cc:348] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.354560 6576 raft_consensus.cc:374] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.354574 6576 raft_consensus.cc:729] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5720a0d2b86242d4b4cfc64c910d0fbf, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.354656 6576 consensus_queue.cc:260] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.354775 6576 ts_tablet_manager.cc:1428] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:40.354831 6582 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.354930 6582 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:40.483175 6586 raft_consensus.cc:482] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:40.483291 6586 raft_consensus.cc:504] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.483541 6586 leader_election.cc:288] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.483819 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a208125ba1784a869b9f0d898da10f53" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b" is_pre_election: true
I20260806 03:42:40.483924 6359 raft_consensus.cc:2454] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.483852 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a208125ba1784a869b9f0d898da10f53" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" is_pre_election: true
I20260806 03:42:40.483987 6501 raft_consensus.cc:2454] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.484135 6150 leader_election.cc:302] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7356fd64d9934cd0800fdb443ca75194, ad6738b639624d09a3323a2c95ae6c2b; no voters:
I20260806 03:42:40.484241 6586 raft_consensus.cc:2781] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:40.484289 6586 raft_consensus.cc:482] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:40.484304 6586 raft_consensus.cc:3037] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.485028 6586 raft_consensus.cc:504] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.485184 6586 leader_election.cc:288] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Requested vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.485332 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a208125ba1784a869b9f0d898da10f53" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b"
I20260806 03:42:40.485352 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a208125ba1784a869b9f0d898da10f53" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
I20260806 03:42:40.485424 6359 raft_consensus.cc:3037] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.485437 6501 raft_consensus.cc:3037] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.486016 6359 raft_consensus.cc:2454] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.486140 6501 raft_consensus.cc:2454] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.486192 6150 leader_election.cc:302] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7356fd64d9934cd0800fdb443ca75194, ad6738b639624d09a3323a2c95ae6c2b; no voters:
I20260806 03:42:40.486284 6586 raft_consensus.cc:2781] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:40.486354 6586 raft_consensus.cc:686] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Becoming Leader. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Running, Role: LEADER
I20260806 03:42:40.486410 6586 consensus_queue.cc:237] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.486752 6583 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: TxnStatusTablet state changed. Reason: New leader 7356fd64d9934cd0800fdb443ca75194. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.486831 6583 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:40.487057 6615 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 03:42:40.487197 6615 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 03:42:40.487145 6059 catalog_manager.cc:5495] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:40.502254 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.499581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37873 (local address 127.0.0.1:33242)
0806 03:42:40.499657 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.499671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.499724 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.499895 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.499896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.499897 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:40.499943 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.499946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.500847 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.500847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.501409 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.501411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.501868 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.501869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.501917 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.501932 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.502105 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.502111 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.502146 (+ 35us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.502195 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 03:42:40.502254 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.499699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.30.24.1:37873)
0806 03:42:40.499764 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.499766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.499775 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:40.499803 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.499804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.499815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.499880 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.500002 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.500772 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.501491 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.501792 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.501863 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.502010 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.502043 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.502104 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.502150 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.502173 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:42:40.503468 6501 raft_consensus.cc:1261] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.503655 6359 raft_consensus.cc:1261] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.503769 6583 consensus_queue.cc:1035] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.503872 6586 consensus_queue.cc:1035] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.504413 6581 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: TxnStatusTablet state changed. Reason: New leader 7356fd64d9934cd0800fdb443ca75194. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.504540 6581 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:40.504591 6582 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: TxnStatusTablet state changed. Reason: New leader 7356fd64d9934cd0800fdb443ca75194. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.504709 6582 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:40.505225 6583 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.505307 6583 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:40.505849 6582 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.505952 6582 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 5720a0d2b86242d4b4cfc64c910d0fbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:40.506721 6586 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.506800 6586 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P 7356fd64d9934cd0800fdb443ca75194: This TxnStatusTablet replica's current role is: LEADER
I20260806 03:42:40.507261 6581 tablet_replica.cc:439] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } } }
I20260806 03:42:40.507344 6581 tablet_replica.cc:442] T a208125ba1784a869b9f0d898da10f53 P ad6738b639624d09a3323a2c95ae6c2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 03:42:40.511722 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.509188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:51323 (local address 127.30.24.1:37873)
0806 03:42:40.509226 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.509228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.509351 (+ 123us) server_negotiation.cc:407] Connection header received
0806 03:42:40.509412 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.509413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.509421 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.509463 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.509574 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.510014 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.511106 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.511283 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.511331 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.511519 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.511544 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.511581 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.511638 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.511653 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:42:40.512010 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.509124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37873 (local address 127.30.24.1:51323)
0806 03:42:40.509282 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.509293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.509349 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.509475 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.509476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.509477 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:40.509524 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.509525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.510091 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.510092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.511014 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.511016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.511353 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.511354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.511414 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.511433 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.511585 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.511586 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.511636 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.511681 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260806 03:42:40.514081 6359 raft_consensus.cc:1261] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.514137 6501 raft_consensus.cc:1261] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.514349 6583 consensus_queue.cc:1035] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.514429 6586 consensus_queue.cc:1035] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.516412 6589 mvcc.cc:205] Tried to move back new op lower bound from 7315405867064291328 to 7315405866238828544. Current Snapshot: MvccSnapshot[applied={T|T < 7315405867064291328}]
I20260806 03:42:40.532516 6058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.532652 6058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.533444 6058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Sending DeleteTablet for 3 replicas of tablet cdc0939591d440f8a65369492e6b26ed
I20260806 03:42:40.533730 6480 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc0939591d440f8a65369492e6b26ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:42468
I20260806 03:42:40.533759 6338 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc0939591d440f8a65369492e6b26ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:49962
I20260806 03:42:40.533828 6600 tablet_replica.cc:330] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: stopping tablet replica
I20260806 03:42:40.533815 6196 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc0939591d440f8a65369492e6b26ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33228
I20260806 03:42:40.533867 6600 raft_consensus.cc:2229] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.533900 6598 tablet_replica.cc:330] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: stopping tablet replica
I20260806 03:42:40.533906 6600 raft_consensus.cc:2258] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.533946 6598 raft_consensus.cc:2229] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:40.533975 6599 tablet_replica.cc:330] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: stopping tablet replica
I20260806 03:42:40.533993 6598 raft_consensus.cc:2258] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.534030 6599 raft_consensus.cc:2229] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.534067 6599 raft_consensus.cc:2258] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.534372 6598 ts_tablet_manager.cc:1905] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.534451 6599 ts_tablet_manager.cc:1905] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.534487 6600 ts_tablet_manager.cc:1905] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.536144 6599 ts_tablet_manager.cc:1918] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:40.536238 6599 log.cc:1196] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/cdc0939591d440f8a65369492e6b26ed
I20260806 03:42:40.536319 6598 ts_tablet_manager.cc:1918] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:40.536392 6598 log.cc:1196] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/cdc0939591d440f8a65369492e6b26ed
I20260806 03:42:40.536490 6599 ts_tablet_manager.cc:1939] T cdc0939591d440f8a65369492e6b26ed P ad6738b639624d09a3323a2c95ae6c2b: Deleting consensus metadata
I20260806 03:42:40.536659 6598 ts_tablet_manager.cc:1939] T cdc0939591d440f8a65369492e6b26ed P 7356fd64d9934cd0800fdb443ca75194: Deleting consensus metadata
I20260806 03:42:40.536871 6600 ts_tablet_manager.cc:1918] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:40.536895 6046 catalog_manager.cc:4841] TS ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431): tablet cdc0939591d440f8a65369492e6b26ed (table test-table [id=f42d026aaa6545999afdf37a27cd54bd]) successfully deleted
I20260806 03:42:40.536932 6600 log.cc:1196] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/cdc0939591d440f8a65369492e6b26ed
I20260806 03:42:40.536970 6046 catalog_manager.cc:4841] TS 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873): tablet cdc0939591d440f8a65369492e6b26ed (table test-table [id=f42d026aaa6545999afdf37a27cd54bd]) successfully deleted
I20260806 03:42:40.537147 6600 ts_tablet_manager.cc:1939] T cdc0939591d440f8a65369492e6b26ed P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting consensus metadata
I20260806 03:42:40.537482 6044 catalog_manager.cc:4841] TS 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089): tablet cdc0939591d440f8a65369492e6b26ed (table test-table [id=f42d026aaa6545999afdf37a27cd54bd]) successfully deleted
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 03:42:40.562427 6126 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:40.556131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.30.24.62:38723)
0806 03:42:40.556201 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.556204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.556301 (+ 97us) server_negotiation.cc:407] Connection header received
0806 03:42:40.556371 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.556372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.556385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.556440 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.557081 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.557830 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.558678 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.558966 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.559044 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.562358 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51556: BlockingRecv error: recv got EOF from 127.0.0.1:51556 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM
I20260806 03:42:40.571707 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.563452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.30.24.62:38723)
0806 03:42:40.563515 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.563517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.563569 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:42:40.563626 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.563627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.563638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.563704 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:40.564155 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.564907 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.565911 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.566214 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.566281 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.568985 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:40.568987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:40.568987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:40.568995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:40.570570 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.570914 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.570915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.570915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.570983 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:40.571194 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:40.571195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:40.571195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:40.571341 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:40.571445 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.571536 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.571618 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:42:40.573400 6058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:40.573588 6058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:40.575140 6338 tablet_service.cc:1467] Processing CreateTablet for tablet 1a79645176ce4af9bc16431480058d3b (DEFAULT_TABLE table=test-table [id=14e01cb455634bf1992df0324f908bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.575274 6480 tablet_service.cc:1467] Processing CreateTablet for tablet 1a79645176ce4af9bc16431480058d3b (DEFAULT_TABLE table=test-table [id=14e01cb455634bf1992df0324f908bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.575316 6338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a79645176ce4af9bc16431480058d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.575395 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a79645176ce4af9bc16431480058d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.576603 6196 tablet_service.cc:1467] Processing CreateTablet for tablet 1a79645176ce4af9bc16431480058d3b (DEFAULT_TABLE table=test-table [id=14e01cb455634bf1992df0324f908bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.576737 6576 tablet_bootstrap.cc:492] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Bootstrap starting.
I20260806 03:42:40.576756 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a79645176ce4af9bc16431480058d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.577410 6576 tablet_bootstrap.cc:654] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.578214 6576 tablet_bootstrap.cc:492] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: No bootstrap required, opened a new log
I20260806 03:42:40.578276 6576 ts_tablet_manager.cc:1397] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 03:42:40.578363 6578 tablet_bootstrap.cc:492] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Bootstrap starting.
I20260806 03:42:40.578446 6576 raft_consensus.cc:348] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.578534 6576 raft_consensus.cc:374] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.578554 6576 raft_consensus.cc:729] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5720a0d2b86242d4b4cfc64c910d0fbf, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.578603 6576 consensus_queue.cc:260] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.578691 6578 tablet_bootstrap.cc:654] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.578732 6576 ts_tablet_manager.cc:1428] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:40.579478 6575 tablet_bootstrap.cc:492] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Bootstrap starting.
I20260806 03:42:40.579640 6578 tablet_bootstrap.cc:492] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: No bootstrap required, opened a new log
I20260806 03:42:40.579684 6578 ts_tablet_manager.cc:1397] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:40.579761 6578 raft_consensus.cc:348] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.579805 6578 raft_consensus.cc:374] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.579819 6578 raft_consensus.cc:729] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.579877 6578 consensus_queue.cc:260] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.579939 6575 tablet_bootstrap.cc:654] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.579964 6578 ts_tablet_manager.cc:1428] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.580777 6575 tablet_bootstrap.cc:492] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: No bootstrap required, opened a new log
I20260806 03:42:40.580834 6575 ts_tablet_manager.cc:1397] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.580993 6575 raft_consensus.cc:348] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.581084 6575 raft_consensus.cc:374] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.581115 6575 raft_consensus.cc:729] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6738b639624d09a3323a2c95ae6c2b, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.581158 6575 consensus_queue.cc:260] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.581269 6575 ts_tablet_manager.cc:1428] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:40.599144 6583 raft_consensus.cc:482] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:40.599241 6583 raft_consensus.cc:504] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.599416 6583 leader_election.cc:288] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.599599 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79645176ce4af9bc16431480058d3b" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b" is_pre_election: true
I20260806 03:42:40.599628 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79645176ce4af9bc16431480058d3b" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" is_pre_election: true
I20260806 03:42:40.599715 6359 raft_consensus.cc:2454] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.599736 6501 raft_consensus.cc:2454] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 0.
I20260806 03:42:40.599917 6148 leader_election.cc:302] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, 7356fd64d9934cd0800fdb443ca75194; no voters:
I20260806 03:42:40.600018 6583 raft_consensus.cc:2781] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:40.600068 6583 raft_consensus.cc:482] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:40.600098 6583 raft_consensus.cc:3037] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.600844 6583 raft_consensus.cc:504] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.600987 6583 leader_election.cc:288] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Requested vote from peers ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431), 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089)
I20260806 03:42:40.601148 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79645176ce4af9bc16431480058d3b" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b"
I20260806 03:42:40.601233 6359 raft_consensus.cc:3037] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.601172 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a79645176ce4af9bc16431480058d3b" candidate_uuid: "7356fd64d9934cd0800fdb443ca75194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf"
I20260806 03:42:40.601270 6501 raft_consensus.cc:3037] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.601926 6501 raft_consensus.cc:2454] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.602042 6359 raft_consensus.cc:2454] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7356fd64d9934cd0800fdb443ca75194 in term 1.
I20260806 03:42:40.602147 6148 leader_election.cc:302] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, 7356fd64d9934cd0800fdb443ca75194; no voters:
I20260806 03:42:40.602268 6583 raft_consensus.cc:2781] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:40.602332 6583 raft_consensus.cc:686] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Becoming Leader. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Running, Role: LEADER
I20260806 03:42:40.602389 6583 consensus_queue.cc:237] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.603003 6058 catalog_manager.cc:5495] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "7356fd64d9934cd0800fdb443ca75194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:40.616143 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.613601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.30.24.1:37873)
0806 03:42:40.613652 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.613654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.613662 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:40.613734 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.613735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.613746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.613822 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.613976 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.614488 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.615390 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.615600 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.615678 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.615837 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.615877 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.615932 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.616006 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.616029 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:42:40.617063 6501 raft_consensus.cc:1261] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.617268 6359 raft_consensus.cc:1261] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Refusing update from remote peer 7356fd64d9934cd0800fdb443ca75194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.617354 6624 consensus_queue.cc:1035] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.617761 6624 consensus_queue.cc:1035] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.619997 6058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.620100 6058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.620812 6058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Sending DeleteTablet for 3 replicas of tablet 1a79645176ce4af9bc16431480058d3b
I20260806 03:42:40.621045 6196 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a79645176ce4af9bc16431480058d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33228
I20260806 03:42:40.621080 6480 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a79645176ce4af9bc16431480058d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:42468
I20260806 03:42:40.621129 6197 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a79645176ce4af9bc16431480058d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33228
I20260806 03:42:40.621160 6600 tablet_replica.cc:330] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: stopping tablet replica
I20260806 03:42:40.621209 6600 raft_consensus.cc:2229] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.621192 6338 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a79645176ce4af9bc16431480058d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:49962
I20260806 03:42:40.621253 6600 raft_consensus.cc:2258] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.621274 6598 tablet_replica.cc:330] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: stopping tablet replica
I20260806 03:42:40.621326 6598 raft_consensus.cc:2229] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:40.621411 6598 raft_consensus.cc:2258] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.621495 6599 tablet_replica.cc:330] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: stopping tablet replica
I20260806 03:42:40.621555 6599 raft_consensus.cc:2229] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.621521 6058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 03:42:40.621610 6599 raft_consensus.cc:2258] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 03:42:40.621659 6058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:40.622020 6599 ts_tablet_manager.cc:1905] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.622025 6600 ts_tablet_manager.cc:1905] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.622108 6598 ts_tablet_manager.cc:1905] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.623799 6599 ts_tablet_manager.cc:1918] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:40.623822 6600 ts_tablet_manager.cc:1918] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:40.623862 6599 log.cc:1196] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/1a79645176ce4af9bc16431480058d3b
I20260806 03:42:40.623880 6600 log.cc:1196] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/1a79645176ce4af9bc16431480058d3b
I20260806 03:42:40.624095 6599 ts_tablet_manager.cc:1939] T 1a79645176ce4af9bc16431480058d3b P ad6738b639624d09a3323a2c95ae6c2b: Deleting consensus metadata
I20260806 03:42:40.624138 6600 ts_tablet_manager.cc:1939] T 1a79645176ce4af9bc16431480058d3b P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting consensus metadata
I20260806 03:42:40.624441 6044 catalog_manager.cc:4841] TS 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089): tablet 1a79645176ce4af9bc16431480058d3b (table test-table [id=14e01cb455634bf1992df0324f908bf6]) successfully deleted
I20260806 03:42:40.624533 6598 ts_tablet_manager.cc:1918] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:42:40.624542 6046 catalog_manager.cc:4841] TS ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431): tablet 1a79645176ce4af9bc16431480058d3b (table test-table [id=14e01cb455634bf1992df0324f908bf6]) successfully deleted
I20260806 03:42:40.624588 6598 log.cc:1196] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/1a79645176ce4af9bc16431480058d3b
I20260806 03:42:40.624812 6598 ts_tablet_manager.cc:1939] T 1a79645176ce4af9bc16431480058d3b P 7356fd64d9934cd0800fdb443ca75194: Deleting consensus metadata
I20260806 03:42:40.625206 6046 catalog_manager.cc:4841] TS 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873): tablet 1a79645176ce4af9bc16431480058d3b (table test-table [id=14e01cb455634bf1992df0324f908bf6]) successfully deleted
I20260806 03:42:40.625309 6480 tablet_service.cc:1467] Processing CreateTablet for tablet f9f39a3a915842eda5180b6402060521 (DEFAULT_TABLE table=test-table [id=d567a5eb80094900a45f9ad802094cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.625427 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f39a3a915842eda5180b6402060521. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:42:40.625490 6046 catalog_manager.cc:4808] TS 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873): delete failed for tablet 1a79645176ce4af9bc16431480058d3b because the tablet was not found. No further retry: Not found: Tablet not found: 1a79645176ce4af9bc16431480058d3b
I20260806 03:42:40.626338 6576 tablet_bootstrap.cc:492] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Bootstrap starting.
I20260806 03:42:40.626384 6338 tablet_service.cc:1467] Processing CreateTablet for tablet f9f39a3a915842eda5180b6402060521 (DEFAULT_TABLE table=test-table [id=d567a5eb80094900a45f9ad802094cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.626478 6338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f39a3a915842eda5180b6402060521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.626729 6576 tablet_bootstrap.cc:654] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.627676 6575 tablet_bootstrap.cc:492] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Bootstrap starting.
I20260806 03:42:40.627727 6197 tablet_service.cc:1467] Processing CreateTablet for tablet f9f39a3a915842eda5180b6402060521 (DEFAULT_TABLE table=test-table [id=d567a5eb80094900a45f9ad802094cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:42:40.627835 6197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f39a3a915842eda5180b6402060521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:40.628283 6575 tablet_bootstrap.cc:654] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.628981 6575 tablet_bootstrap.cc:492] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: No bootstrap required, opened a new log
I20260806 03:42:40.629030 6575 ts_tablet_manager.cc:1397] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:40.629068 6578 tablet_bootstrap.cc:492] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194: Bootstrap starting.
I20260806 03:42:40.629170 6575 raft_consensus.cc:348] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.629241 6575 raft_consensus.cc:374] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.629261 6575 raft_consensus.cc:729] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6738b639624d09a3323a2c95ae6c2b, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.629305 6575 consensus_queue.cc:260] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.629398 6575 ts_tablet_manager.cc:1428] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.629528 6578 tablet_bootstrap.cc:654] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:40.630228 6576 tablet_bootstrap.cc:492] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: No bootstrap required, opened a new log
I20260806 03:42:40.630283 6576 ts_tablet_manager.cc:1397] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260806 03:42:40.630332 6578 tablet_bootstrap.cc:492] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194: No bootstrap required, opened a new log
I20260806 03:42:40.630375 6578 ts_tablet_manager.cc:1397] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:40.630439 6576 raft_consensus.cc:348] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.630520 6576 raft_consensus.cc:374] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.630543 6576 raft_consensus.cc:729] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5720a0d2b86242d4b4cfc64c910d0fbf, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.630518 6578 raft_consensus.cc:348] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.630595 6578 raft_consensus.cc:374] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:40.630620 6578 raft_consensus.cc:729] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7356fd64d9934cd0800fdb443ca75194, State: Initialized, Role: FOLLOWER
I20260806 03:42:40.630604 6576 consensus_queue.cc:260] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.630663 6578 consensus_queue.cc:260] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.630695 6576 ts_tablet_manager.cc:1428] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:40.630762 6578 ts_tablet_manager.cc:1428] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 03:42:40.666947 6582 raft_consensus.cc:482] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:40.667052 6582 raft_consensus.cc:504] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.667441 6582 leader_election.cc:288] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873), ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431)
I20260806 03:42:40.670644 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.667892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:45387 (local address 127.30.24.2:44431)
0806 03:42:40.667934 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.667936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.668017 (+ 81us) server_negotiation.cc:407] Connection header received
0806 03:42:40.668096 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.668097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.668110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.668170 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.668381 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.668877 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.669897 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.670260 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.670313 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.670322 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.670498 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.670549 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:42:40.670661 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.667815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:44431 (local address 127.30.24.3:45387)
0806 03:42:40.667939 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.667954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.668046 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.668200 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.668201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.668203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.668302 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.668304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.669004 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.669005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.669804 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.669806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.670358 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.670360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.670420 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.670425 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.670475 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.670506 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":17,"thread_start_us":63,"threads_started":1}
I20260806 03:42:40.670720 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.667707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52755 (local address 127.30.24.1:37873)
0806 03:42:40.667779 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.667781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.667866 (+ 85us) server_negotiation.cc:407] Connection header received
0806 03:42:40.667937 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.667938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.667953 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.668007 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:42:40.668267 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.668858 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.669941 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.670313 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.670387 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.670401 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.670560 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.670615 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:42:40.670790 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.667603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37873 (local address 127.30.24.3:52755)
0806 03:42:40.667766 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.667779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.667861 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.668087 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.668088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.668091 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:42:40.668176 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.668178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.668935 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.668936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.669815 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.669818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.670405 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.670406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.670476 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.670482 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.670539 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.670591 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":46,"thread_start_us":99,"threads_started":1}
I20260806 03:42:40.671080 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f39a3a915842eda5180b6402060521" candidate_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b" is_pre_election: true
I20260806 03:42:40.671046 6217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f39a3a915842eda5180b6402060521" candidate_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7356fd64d9934cd0800fdb443ca75194" is_pre_election: true
I20260806 03:42:40.671180 6359 raft_consensus.cc:2454] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5720a0d2b86242d4b4cfc64c910d0fbf in term 0.
I20260806 03:42:40.671198 6217 raft_consensus.cc:2454] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5720a0d2b86242d4b4cfc64c910d0fbf in term 0.
I20260806 03:42:40.671360 6434 leader_election.cc:302] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, ad6738b639624d09a3323a2c95ae6c2b; no voters:
I20260806 03:42:40.671499 6582 raft_consensus.cc:2781] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:40.671542 6582 raft_consensus.cc:482] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:40.671568 6582 raft_consensus.cc:3037] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.672282 6582 raft_consensus.cc:504] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.672427 6582 leader_election.cc:288] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [CANDIDATE]: Term 1 election: Requested vote from peers 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873), ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431)
I20260806 03:42:40.672618 6217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f39a3a915842eda5180b6402060521" candidate_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7356fd64d9934cd0800fdb443ca75194"
I20260806 03:42:40.672653 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f39a3a915842eda5180b6402060521" candidate_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6738b639624d09a3323a2c95ae6c2b"
I20260806 03:42:40.672704 6217 raft_consensus.cc:3037] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.672713 6359 raft_consensus.cc:3037] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:40.673475 6359 raft_consensus.cc:2454] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5720a0d2b86242d4b4cfc64c910d0fbf in term 1.
I20260806 03:42:40.673501 6217 raft_consensus.cc:2454] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5720a0d2b86242d4b4cfc64c910d0fbf in term 1.
I20260806 03:42:40.673642 6434 leader_election.cc:302] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5720a0d2b86242d4b4cfc64c910d0fbf, ad6738b639624d09a3323a2c95ae6c2b; no voters:
I20260806 03:42:40.673772 6582 raft_consensus.cc:2781] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:40.673898 6582 raft_consensus.cc:686] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 LEADER]: Becoming Leader. State: Replica: 5720a0d2b86242d4b4cfc64c910d0fbf, State: Running, Role: LEADER
I20260806 03:42:40.673976 6582 consensus_queue.cc:237] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } }
I20260806 03:42:40.674718 6058 catalog_manager.cc:5495] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3). New cstate: current_term: 1 leader_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5720a0d2b86242d4b4cfc64c910d0fbf" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37089 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:40.690479 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.688269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.30.24.3:37089)
0806 03:42:40.688313 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.688315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.688362 (+ 47us) server_negotiation.cc:407] Connection header received
0806 03:42:40.688426 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.688426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.688444 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.688567 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.688667 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.689119 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.689792 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.689968 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.690052 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.690194 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.690312 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.690371 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.690408 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.690425 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:42:40.695322 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.692853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:43719 (local address 127.30.24.1:37873)
0806 03:42:40.692950 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.692952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.693079 (+ 127us) server_negotiation.cc:407] Connection header received
0806 03:42:40.693147 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.693148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.693159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.693239 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.693380 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.693834 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.694619 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.694814 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.694892 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.695065 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.695105 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.695168 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.695232 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.695250 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 03:42:40.695362 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.692771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37873 (local address 127.30.24.3:43719)
0806 03:42:40.692999 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.693009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.693096 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.693243 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.693244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.693246 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:40.693304 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.693305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.693922 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.693923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.694497 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.694499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.694893 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.694894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.694960 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.694981 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.695172 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.695174 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.695227 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.695271 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":131,"threads_started":1}
I20260806 03:42:40.697781 6359 raft_consensus.cc:1261] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Refusing update from remote peer 5720a0d2b86242d4b4cfc64c910d0fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.697870 6217 raft_consensus.cc:1261] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Refusing update from remote peer 5720a0d2b86242d4b4cfc64c910d0fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.698103 6582 consensus_queue.cc:1035] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6738b639624d09a3323a2c95ae6c2b" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.698272 6651 consensus_queue.cc:1035] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7356fd64d9934cd0800fdb443ca75194" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:40.705592 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.703273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:35233 (local address 127.30.24.3:37089)
0806 03:42:40.703319 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:40.703321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:40.703330 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:40.703401 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:40.703401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:40.703412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:40.703471 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:42:40.703578 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.704028 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.704778 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.705022 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.705108 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.705318 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.705377 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.705439 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:40.705504 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 03:42:40.705523 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 03:42:40.705626 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.703241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37089 (local address 127.30.24.1:35233)
0806 03:42:40.703280 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.703297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:40.703336 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:40.703476 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:40.703477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:40.703478 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 03:42:40.703519 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.703520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.704112 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.704114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.704689 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:40.704691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:40.705110 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:40.705111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:40.705168 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:40.705175 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 03:42:40.705441 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 03:42:40.705444 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:40.705498 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 03:42:40.705550 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260806 03:42:40.712314 6058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.712415 6058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:42:40.713090 6058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34915d19cdf44fcd9d35e7996d9f358e: Sending DeleteTablet for 3 replicas of tablet f9f39a3a915842eda5180b6402060521
I20260806 03:42:40.713385 6197 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f39a3a915842eda5180b6402060521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33228
I20260806 03:42:40.713393 6480 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f39a3a915842eda5180b6402060521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:42468
I20260806 03:42:40.713480 6600 tablet_replica.cc:330] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: stopping tablet replica
I20260806 03:42:40.713485 6338 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f39a3a915842eda5180b6402060521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:42:40 UTC) from {username='oryx', principal='oryx/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:49962
I20260806 03:42:40.713527 6600 raft_consensus.cc:2229] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:40.713577 6598 tablet_replica.cc:330] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194: stopping tablet replica
I20260806 03:42:40.713613 6600 raft_consensus.cc:2258] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.713637 6598 raft_consensus.cc:2229] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.713690 6598 raft_consensus.cc:2258] T f9f39a3a915842eda5180b6402060521 P 7356fd64d9934cd0800fdb443ca75194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.713753 6599 tablet_replica.cc:330] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: stopping tablet replica
I20260806 03:42:40.713806 6599 raft_consensus.cc:2229] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:40.713845 6599 raft_consensus.cc:2258] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:40.714013 6600 ts_tablet_manager.cc:1905] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.714287 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6131
I20260806 03:42:40.716830 6600 ts_tablet_manager.cc:1918] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:40.716897 6600 log.cc:1196] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/f9f39a3a915842eda5180b6402060521
I20260806 03:42:40.717127 6600 ts_tablet_manager.cc:1939] T f9f39a3a915842eda5180b6402060521 P 5720a0d2b86242d4b4cfc64c910d0fbf: Deleting consensus metadata
I20260806 03:42:40.717433 6599 ts_tablet_manager.cc:1905] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:42:40.717666 6044 catalog_manager.cc:4841] TS 5720a0d2b86242d4b4cfc64c910d0fbf (127.30.24.3:37089): tablet f9f39a3a915842eda5180b6402060521 (table test-table [id=d567a5eb80094900a45f9ad802094cdd]) successfully deleted
I20260806 03:42:40.718653 6599 ts_tablet_manager.cc:1918] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 03:42:40.718724 6599 log.cc:1196] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/f9f39a3a915842eda5180b6402060521
I20260806 03:42:40.718945 6599 ts_tablet_manager.cc:1939] T f9f39a3a915842eda5180b6402060521 P ad6738b639624d09a3323a2c95ae6c2b: Deleting consensus metadata
I20260806 03:42:40.719350 6046 catalog_manager.cc:4841] TS ad6738b639624d09a3323a2c95ae6c2b (127.30.24.2:44431): tablet f9f39a3a915842eda5180b6402060521 (table test-table [id=d567a5eb80094900a45f9ad802094cdd]) successfully deleted
W20260806 03:42:40.725238 6046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:37873 (error 108)
I20260806 03:42:40.725498 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6273
I20260806 03:42:40.726197 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:40.725639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37873 (local address 127.0.0.1:33254)
0806 03:42:40.726088 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:40.726134 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":45,"threads_started":1}
W20260806 03:42:40.726282 6046 catalog_manager.cc:4584] TS 7356fd64d9934cd0800fdb443ca75194 (127.30.24.1:37873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9f39a3a915842eda5180b6402060521: Network error: Client connection negotiation failed: client connection to 127.30.24.1:37873: connect: Connection refused (error 111)
I20260806 03:42:40.732542 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6414
I20260806 03:42:40.738751 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6027
2026-08-06T03:42:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2482 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6664](info): set up 2 sockets
Aug 06 03:42:40 dist-test-slave-10pc krb5kdc[6664](info): commencing operation
krb5kdc: starting...
W20260806 03:42:41.802469 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:41 dist-test-slave-10pc krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:41.925362 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42267
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:42.010135 6677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:42.010352 6677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:42.010375 6677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:42.011495 6677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:42.011533 6677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:42.011548 6677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:42.011560 6677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:42.011572 6677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:42.013036 6677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:42267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:42.013248 6677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:42.013437 6677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:42.018486 6677 server_base.cc:1034] running on GCE node
W20260806 03:42:42.018458 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:42.018469 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:42.018469 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:42.019047 6677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:42.019332 6677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:42.020551 6677 hybrid_clock.cc:648] HybridClock initialized: now 1785987762020535 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T03:42:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1282 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:42:42 dist-test-slave-10pc krb5kdc[6690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:42:42 dist-test-slave-10pc krb5kdc[6690](info): set up 2 sockets
Aug 06 03:42:42 dist-test-slave-10pc krb5kdc[6690](info): commencing operation
krb5kdc: starting...
W20260806 03:42:43.094125 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:42:43.220202 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40855
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:43.304988 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:43.305218 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:43.305243 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:43.306442 6703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:43.306485 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:43.306499 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:43.306512 6703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:43.306558 6703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:43.308228 6703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:43.308548 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:43.308800 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:43.314308 6703 server_base.cc:1034] running on GCE node
W20260806 03:42:43.314289 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.314294 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.314270 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:43.314776 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:43.315038 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:43.316184 6703 hybrid_clock.cc:648] HybridClock initialized: now 1785987763316153 us; error 61 us; skew 500 ppm
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:43.323343 6703 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:43.323823 6703 webserver.cc:460] Webserver started at http://127.30.24.62:32789/ using document root <none> and password file <none>
I20260806 03:42:43.324067 6703 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:43.324115 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:43.324268 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:43.325465 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "baa4e60c8519407b89c05306b0a71a99"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "7bdaeb5bad40a6fc9031094d5c0191e8"
server_key_iv: "263a65b6a6eef5e6a8c5d9fb14bea330"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.325852 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "baa4e60c8519407b89c05306b0a71a99"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "7bdaeb5bad40a6fc9031094d5c0191e8"
server_key_iv: "263a65b6a6eef5e6a8c5d9fb14bea330"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.327248 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:42:43.328006 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.328243 6703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:43.328353 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "baa4e60c8519407b89c05306b0a71a99"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "7bdaeb5bad40a6fc9031094d5c0191e8"
server_key_iv: "263a65b6a6eef5e6a8c5d9fb14bea330"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.328440 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:43.340746 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:43.342967 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:43.343116 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:43.347456 6703 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:40855
I20260806 03:42:43.347554 6770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:40855 every 8 connection(s)
I20260806 03:42:43.347901 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:43.348821 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:43.351138 6771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Bootstrap starting.
I20260806 03:42:43.351722 6771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:43.351963 6771 log.cc:824] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:43.352772 6771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: No bootstrap required, opened a new log
I20260806 03:42:43.354425 6771 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:43.354571 6771 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:43.354595 6771 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa4e60c8519407b89c05306b0a71a99, State: Initialized, Role: FOLLOWER
I20260806 03:42:43.354703 6771 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:43.354784 6771 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:43.354823 6771 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:43.354890 6771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:43.355511 6771 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:43.355638 6771 leader_election.cc:302] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa4e60c8519407b89c05306b0a71a99; no voters:
I20260806 03:42:43.355882 6771 leader_election.cc:288] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:43.355924 6774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:43.356040 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6703
I20260806 03:42:43.356169 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:43.356159 6774 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 LEADER]: Becoming Leader. State: Replica: baa4e60c8519407b89c05306b0a71a99, State: Running, Role: LEADER
I20260806 03:42:43.356213 6771 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:43.356253 6774 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:43.356395 30816 external_mini_cluster.cc:1428] Setting key 51f0c171876a8cd6ba1b2367762bbbc2
I20260806 03:42:43.356596 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa4e60c8519407b89c05306b0a71a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } } }
I20260806 03:42:43.356679 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:43.356819 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa4e60c8519407b89c05306b0a71a99. Latest consensus state: current_term: 1 leader_uuid: "baa4e60c8519407b89c05306b0a71a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } } }
I20260806 03:42:43.356866 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:43.356982 6782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:43.357731 6782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:43.359512 6782 catalog_manager.cc:1347] Generated new cluster ID: 0595050c3464414c91e13ee52a233fd3
I20260806 03:42:43.359580 6782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:43.367962 6782 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:43.368734 6782 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:43.369508 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.357354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.30.24.62:40855)
0806 03:42:43.357592 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:43.357602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:43.357649 (+ 47us) server_negotiation.cc:407] Connection header received
0806 03:42:43.357783 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:43.357792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:43.357869 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:43.358800 (+ 931us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:43.358921 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.360302 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.361417 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.362025 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.362124 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.365098 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:43.365103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:43.365103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:43.365120 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:43.367529 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.368075 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.368079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.368079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.368215 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.368475 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.368476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.368477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.368780 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:43.369029 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:43.369107 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 03:42:43.369271 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260806 03:42:43.380829 6782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Generated new TSK 0
I20260806 03:42:43.381034 6782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:42:43.407418 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40855
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:43.492388 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:43.492717 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:43.492743 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:43.493963 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:43.494045 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:43.494076 6800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:43.495519 6800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:43.495713 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:43.495909 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:42:43.501077 6800 server_base.cc:1034] running on GCE node
W20260806 03:42:43.500926 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.500988 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.500931 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:43.501480 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:43.501749 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:43.502951 6800 hybrid_clock.cc:648] HybridClock initialized: now 1785987763502943 us; error 35 us; skew 500 ppm
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:43.506301 6800 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:42:43.506749 6800 webserver.cc:460] Webserver started at http://127.30.24.1:34519/ using document root <none> and password file <none>
I20260806 03:42:43.507002 6800 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:43.507069 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:43.507208 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:43.508288 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "ceb74d063c494e6686577d810c467fc7"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "12e58c0f3b1d85026c18a6e4f0b6fe24"
server_key_iv: "1f8b04e9ec5a0ae29ed5b29f44ef6226"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.508695 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb74d063c494e6686577d810c467fc7"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "12e58c0f3b1d85026c18a6e4f0b6fe24"
server_key_iv: "1f8b04e9ec5a0ae29ed5b29f44ef6226"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.510035 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:43.510738 6816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.510907 6800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:43.511006 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "ceb74d063c494e6686577d810c467fc7"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "12e58c0f3b1d85026c18a6e4f0b6fe24"
server_key_iv: "1f8b04e9ec5a0ae29ed5b29f44ef6226"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.511077 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:43.521718 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:43.523641 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:43.523804 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:43.524061 6800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:43.524391 6800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:43.524442 6800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.524505 6800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:43.524536 6800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.530186 6800 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:38197
I20260806 03:42:43.530229 6933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:38197 every 8 connection(s)
I20260806 03:42:43.530583 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:43.533001 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6800
I20260806 03:42:43.533097 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:43.533295 30816 external_mini_cluster.cc:1428] Setting key 38cfa6251137af2846328cceda9cd40e
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:43.541329 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.531392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:42715 (local address 127.30.24.62:40855)
0806 03:42:43.531450 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:43.531452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:43.531665 (+ 213us) server_negotiation.cc:407] Connection header received
0806 03:42:43.532199 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:43.532200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:43.532234 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:43.532288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:43.533410 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.534104 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.535115 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.535462 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.535538 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.538369 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:43.538370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:43.538370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:43.538378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:43.539866 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.540291 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.540293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.540293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.540356 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.540656 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.540657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.540657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.540776 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:43.540844 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:43.541133 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 03:42:43.541235 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 03:42:43.541486 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.531388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:42715)
0806 03:42:43.531542 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:43.531578 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:43.532093 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:43.532365 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:43.532368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:43.533021 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:43.533289 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:43.533295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.534202 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.534203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:43.535029 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:43.535032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.535550 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.535551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:43.535729 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.536269 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:43.536281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:43.538257 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:43.539944 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:43.539951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:43.539952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:43.540207 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:43.540424 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:43.540425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:43.540425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:43.540582 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:43.540847 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:43.540855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:43.540992 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 03:42:43.541089 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 03:42:43.541252 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260806 03:42:43.542158 6934 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40855
I20260806 03:42:43.542280 6934 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:43.542497 6934 heartbeater.cc:511] Master 127.30.24.62:40855 requested a full tablet report, sending...
I20260806 03:42:43.542995 6735 ts_manager.cc:194] Registered new tserver with Master: ceb74d063c494e6686577d810c467fc7 (127.30.24.1:38197)
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
I20260806 03:42:43.543627 6735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:42715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:42:43.563371 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40855
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:43.648342 6940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:43.648667 6940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:43.648696 6940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:43.649876 6940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:43.649912 6940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:43.649938 6940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:43.651365 6940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:43.651582 6940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:43.651787 6940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:43.657135 6946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.657200 6945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.657145 6948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:43.657462 6940 server_base.cc:1034] running on GCE node
I20260806 03:42:43.657670 6940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:43.657932 6940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:43.659075 6940 hybrid_clock.cc:648] HybridClock initialized: now 1785987763659062 us; error 28 us; skew 500 ppm
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:43.661687 6940 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:42:43.662117 6940 webserver.cc:460] Webserver started at http://127.30.24.2:42443/ using document root <none> and password file <none>
I20260806 03:42:43.662322 6940 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:43.662362 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:43.662454 6940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:43.663501 6940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "6e48bd56b5214de981ee3a975ebdc85d"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "fbc63c20fd8bf837b1371afd06829e8e"
server_key_iv: "a45139abd0f8d027eb5034008c7917d2"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.663841 6940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "6e48bd56b5214de981ee3a975ebdc85d"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "fbc63c20fd8bf837b1371afd06829e8e"
server_key_iv: "a45139abd0f8d027eb5034008c7917d2"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.665256 6940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:43.666029 6955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.666213 6940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:43.666297 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "6e48bd56b5214de981ee3a975ebdc85d"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "fbc63c20fd8bf837b1371afd06829e8e"
server_key_iv: "a45139abd0f8d027eb5034008c7917d2"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.666355 6940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:43.679366 6940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:43.681687 6940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:43.681803 6940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:43.681985 6940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:43.682299 6940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:43.682335 6940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.682375 6940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:43.682402 6940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.688241 6940 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:44475
I20260806 03:42:43.688328 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:44475 every 8 connection(s)
I20260806 03:42:43.688658 6940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:43.689126 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 6940
I20260806 03:42:43.689261 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:43.689473 30816 external_mini_cluster.cc:1428] Setting key d1ec160ad7a1d21d9b1d30d72ca8b4a4
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
I20260806 03:42:43.699065 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.689489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:52305 (local address 127.30.24.62:40855)
0806 03:42:43.689560 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:43.689563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:43.690076 (+ 513us) server_negotiation.cc:407] Connection header received
0806 03:42:43.690520 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:43.690521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:43.690532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:43.690574 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:43.691907 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.692411 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.693492 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.693695 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.693769 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.696029 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:43.696030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:43.696031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:43.696039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:43.697577 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.698012 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.698014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.698014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.698073 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.698387 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.698389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.698390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.698557 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:43.698647 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:43.698921 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 03:42:43.698996 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:42:43.699352 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.689463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:52305)
0806 03:42:43.689669 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:43.689727 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:43.690407 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:43.690645 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:43.690650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:43.691503 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:43.691794 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:43.691801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.692527 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.692529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:43.693384 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:43.693387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.693777 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.693778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:43.693904 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.694534 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:43.694542 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:43.695931 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:43.697685 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:43.697693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:43.697696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:43.697924 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:43.698141 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:43.698142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:43.698143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:43.698256 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:43.698654 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:43.698661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:43.698796 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 03:42:43.698893 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 03:42:43.699029 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260806 03:42:43.699903 7073 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40855
I20260806 03:42:43.700014 7073 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:43.700208 7073 heartbeater.cc:511] Master 127.30.24.62:40855 requested a full tablet report, sending...
I20260806 03:42:43.700630 6735 ts_manager.cc:194] Registered new tserver with Master: 6e48bd56b5214de981ee3a975ebdc85d (127.30.24.2:44475)
I20260806 03:42:43.701040 6735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:52305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:42:43.717410 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40855
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:43.801239 7078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:43.801445 7078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:43.801468 7078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:43.802635 7078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:43.802676 7078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:43.802702 7078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:43.804078 7078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:43.804270 7078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:43.804474 7078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:43.809449 7086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:43.809513 7078 server_base.cc:1034] running on GCE node
W20260806 03:42:43.809516 7083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:43.809468 7084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:43.809859 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:43.810058 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:43.811194 7078 hybrid_clock.cc:648] HybridClock initialized: now 1785987763811177 us; error 34 us; skew 500 ppm
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:43.813788 7078 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:42:43.814268 7078 webserver.cc:460] Webserver started at http://127.30.24.3:34529/ using document root <none> and password file <none>
I20260806 03:42:43.814515 7078 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:43.814594 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:43.814721 7078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:43.815747 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "b7c4c220c77244eeb1be3008eff3703c"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "4888019e868196796572a99c7d4ef15c"
server_key_iv: "b2065f275b6bf272c9eaf4529cb51bb2"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.816061 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "b7c4c220c77244eeb1be3008eff3703c"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "4888019e868196796572a99c7d4ef15c"
server_key_iv: "b2065f275b6bf272c9eaf4529cb51bb2"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.817476 7078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:42:43.818310 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.818487 7078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:43.818602 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "b7c4c220c77244eeb1be3008eff3703c"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "4888019e868196796572a99c7d4ef15c"
server_key_iv: "b2065f275b6bf272c9eaf4529cb51bb2"
server_key_version: "encryptionkey@0"
I20260806 03:42:43.818691 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:43.837561 7078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:43.839738 7078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:43.839927 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:43.840188 7078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:43.840566 7078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:43.840624 7078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.840658 7078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:43.840684 7078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:43.846690 7078 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:44921
I20260806 03:42:43.846731 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:44921 every 8 connection(s)
I20260806 03:42:43.847072 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:42:43.852965 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7078
I20260806 03:42:43.853099 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
Aug 06 03:42:43 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:42:43.853348 30816 external_mini_cluster.cc:1428] Setting key 62a22bb4acabbc534f5883b65764db76
I20260806 03:42:43.856664 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.847890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:58391 (local address 127.30.24.62:40855)
0806 03:42:43.847949 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:43.847951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:43.848106 (+ 155us) server_negotiation.cc:407] Connection header received
0806 03:42:43.848661 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:43.848662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:43.848673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:43.848747 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:43.849692 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.850197 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.851176 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.851363 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.851421 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.853760 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:43.853762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:43.853762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:43.853769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:43.855269 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.855704 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.855705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.855706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.855767 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:43.856003 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:43.856003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:43.856004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:43.856123 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:43.856179 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:43.856517 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 03:42:43.856599 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:42:43.856936 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:43.847807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:58391)
0806 03:42:43.847963 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:43.848006 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:43.848565 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:43.848802 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:43.848805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:43.849374 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:43.849580 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:43.849585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.850284 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.850285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:43.851092 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:43.851094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:43.851438 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:43.851439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:43.851538 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:43.852051 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:43.852060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:43.853650 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:43.855341 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:43.855348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:43.855350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:43.855587 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:43.855842 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:42:43.855843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:42:43.855843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:42:43.855947 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:42:43.856180 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:43.856185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:43.856362 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 03:42:43.856502 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 03:42:43.856653 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260806 03:42:43.857481 7210 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40855
I20260806 03:42:43.857575 7210 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:43.857724 7210 heartbeater.cc:511] Master 127.30.24.62:40855 requested a full tablet report, sending...
I20260806 03:42:43.858114 6735 ts_manager.cc:194] Registered new tserver with Master: b7c4c220c77244eeb1be3008eff3703c (127.30.24.3:44921)
I20260806 03:42:43.858484 6735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:58391
I20260806 03:42:43.865249 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab.
Entry for principal oryx/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab.
WARNING: no policy specified for oryx/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab.
Entry for principal oryx/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab.
WARNING: no policy specified for oryx/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab.
Entry for principal oryx/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab.
I20260806 03:42:43.947088 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6800
I20260806 03:42:43.953171 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6940
I20260806 03:42:43.958848 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7078
I20260806 03:42:43.963899 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 6703
I20260806 03:42:43.969777 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40855
--webserver_interface=127.30.24.62
--webserver_port=32789
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:42:44.054823 7219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:44.055029 7219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:44.055051 7219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:44.056140 7219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:44.056183 7219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:44.056197 7219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:44.056210 7219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:44.056222 7219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:44.057665 7219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:44.057871 7219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:44.058051 7219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:44.063421 7225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:44.063416 7224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.063508 7219 server_base.cc:1034] running on GCE node
W20260806 03:42:44.063416 7227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.064000 7219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:44.064276 7219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:44.065441 7219 hybrid_clock.cc:648] HybridClock initialized: now 1785987764065423 us; error 38 us; skew 500 ppm
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:44.068058 7219 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:42:44.068646 7219 webserver.cc:460] Webserver started at http://127.30.24.62:32789/ using document root <none> and password file <none>
I20260806 03:42:44.068922 7219 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:44.068987 7219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:44.070502 7219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:44.071199 7235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.071364 7219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:44.071448 7219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "baa4e60c8519407b89c05306b0a71a99"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "7bdaeb5bad40a6fc9031094d5c0191e8"
server_key_iv: "263a65b6a6eef5e6a8c5d9fb14bea330"
server_key_version: "encryptionkey@0"
I20260806 03:42:44.071817 7219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:44.086632 7219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:44.089103 7219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:44.089233 7219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:44.093235 7219 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:40855
I20260806 03:42:44.093271 7287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:40855 every 8 connection(s)
I20260806 03:42:44.093693 7219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:44.095045 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7219
I20260806 03:42:44.095384 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:38197
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:40855
--webserver_port=34519
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:42:44.095479 7288 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:42:44.096642 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Bootstrap starting.
I20260806 03:42:44.105526 7288 log.cc:824] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:44.107573 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:44.107913 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Bootstrap complete.
I20260806 03:42:44.110466 7288 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:44.110720 7288 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa4e60c8519407b89c05306b0a71a99, State: Initialized, Role: FOLLOWER
I20260806 03:42:44.110898 7288 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:44.110970 7288 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:44.111023 7288 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:44.111073 7288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:44.112113 7288 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:44.112228 7288 leader_election.cc:302] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa4e60c8519407b89c05306b0a71a99; no voters:
I20260806 03:42:44.112440 7288 leader_election.cc:288] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:42:44.112509 7292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:44.112645 7292 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [term 2 LEADER]: Becoming Leader. State: Replica: baa4e60c8519407b89c05306b0a71a99, State: Running, Role: LEADER
I20260806 03:42:44.112757 7288 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:44.112733 7292 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } }
I20260806 03:42:44.113101 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa4e60c8519407b89c05306b0a71a99. Latest consensus state: current_term: 2 leader_uuid: "baa4e60c8519407b89c05306b0a71a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } } }
I20260806 03:42:44.113178 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:44.113453 7297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:44.113412 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "baa4e60c8519407b89c05306b0a71a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa4e60c8519407b89c05306b0a71a99" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40855 } } }
I20260806 03:42:44.113617 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:44.113998 7297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:44.114223 7297 catalog_manager.cc:1259] Loaded cluster ID: 0595050c3464414c91e13ee52a233fd3
I20260806 03:42:44.114408 7297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:44.115674 7297 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:44.115870 7297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P baa4e60c8519407b89c05306b0a71a99: Loaded TSK: 0
I20260806 03:42:44.116088 7297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:42:44.182984 7290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:44.183235 7290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:44.183280 7290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:44.184523 7290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:44.184582 7290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:44.184628 7290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:44.186123 7290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.1.keytab
--principal=oryx/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:38197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=34519
--tserver_master_addrs=127.30.24.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:44.186393 7290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:44.186635 7290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:44.191685 7315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:44.191787 7318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:44.191865 7316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.192337 7290 server_base.cc:1034] running on GCE node
I20260806 03:42:44.192600 7290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:44.192881 7290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:44.194063 7290 hybrid_clock.cc:648] HybridClock initialized: now 1785987764194031 us; error 47 us; skew 500 ppm
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987764, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:44.196998 7290 init.cc:377] Logged in from keytab as oryx/127.30.24.1@KRBTEST.COM (short username oryx)
I20260806 03:42:44.197570 7290 webserver.cc:460] Webserver started at http://127.30.24.1:34519/ using document root <none> and password file <none>
I20260806 03:42:44.197839 7290 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:44.197925 7290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:44.199774 7290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 03:42:44.200538 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.200749 7290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:44.200853 7290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "ceb74d063c494e6686577d810c467fc7"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "12e58c0f3b1d85026c18a6e4f0b6fe24"
server_key_iv: "1f8b04e9ec5a0ae29ed5b29f44ef6226"
server_key_version: "encryptionkey@0"
I20260806 03:42:44.201320 7290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:44.221238 7290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:44.223930 7290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:44.224104 7290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:44.224345 7290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:44.224730 7290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:44.224793 7290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.224838 7290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:44.224865 7290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.230847 7290 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:38197
I20260806 03:42:44.230896 7441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:38197 every 8 connection(s)
I20260806 03:42:44.231251 7290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:44.231840 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7290
I20260806 03:42:44.232182 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:44475
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:40855
--webserver_port=42443
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.243295 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.232089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:42147)
0806 03:42:44.232513 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.232573 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.233213 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.233904 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.233921 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.234696 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.234979 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.234986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.236775 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.236777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.238114 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.238118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.239098 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.239099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.239274 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.239939 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.239950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.242988 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":88,"threads_started":1}
W20260806 03:42:44.243788 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.232440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:42147 (local address 127.30.24.62:40855)
0806 03:42:44.233431 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.233440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.233477 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:42:44.233616 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.233623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.233678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.235118 (+ 1440us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.235202 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.236661 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.238262 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.238973 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.239577 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.243567 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:42147: BlockingRecv error: recv got EOF from 127.30.24.1:42147 (error 108)
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":74,"threads_started":1}
W20260806 03:42:44.244155 7442 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:40855 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.24.62:40855: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.250222 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.244480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:38643)
0806 03:42:44.244522 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.244532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.244570 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.244737 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.244738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.244891 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.244946 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.244948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.245924 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.245926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.246831 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.246834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.247328 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.247329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.247410 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.247913 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.247920 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.250151 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 03:42:44.250551 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.244549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:38643 (local address 127.30.24.62:40855)
0806 03:42:44.244588 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.244590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.244601 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:44.244626 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.244627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.244638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.245038 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.245071 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.245827 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.246949 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.247242 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.247315 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.250473 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:38643: BlockingRecv error: recv got EOF from 127.30.24.1:38643 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.255865 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.250504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:39001)
0806 03:42:44.250729 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.250741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.250792 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.250938 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.250939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.251082 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.251118 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.251120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.251950 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.251951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.252799 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.252801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.253240 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.253241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.253295 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.253754 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.253759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.255789 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209}
W20260806 03:42:44.255951 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.250567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:39001 (local address 127.30.24.62:40855)
0806 03:42:44.250661 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.250663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.250778 (+ 115us) server_negotiation.cc:407] Connection header received
0806 03:42:44.250848 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.250849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.250859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.250922 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.251204 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.251870 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.252873 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.253167 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.253234 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.255886 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:39001: BlockingRecv error: recv got EOF from 127.30.24.1:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":129,"threads_started":1}
W20260806 03:42:44.256234 7442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:42:44.327678 7446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:44.327946 7446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:44.327988 7446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:44.329344 7446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:44.329450 7446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:44.329504 7446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:44.331357 7446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.2.keytab
--principal=oryx/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:44475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=42443
--tserver_master_addrs=127.30.24.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:44.331684 7446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:44.331943 7446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:44.337051 7453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.337227 7446 server_base.cc:1034] running on GCE node
W20260806 03:42:44.337060 7455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:44.337060 7452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.337543 7446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:44.337814 7446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:44.339046 7446 hybrid_clock.cc:648] HybridClock initialized: now 1785987764339031 us; error 40 us; skew 500 ppm
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987764, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:44.341845 7446 init.cc:377] Logged in from keytab as oryx/127.30.24.2@KRBTEST.COM (short username oryx)
I20260806 03:42:44.342358 7446 webserver.cc:460] Webserver started at http://127.30.24.2:42443/ using document root <none> and password file <none>
I20260806 03:42:44.342615 7446 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:44.342741 7446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:44.344152 7446 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:44.344799 7462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.344964 7446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:44.345057 7446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "6e48bd56b5214de981ee3a975ebdc85d"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "fbc63c20fd8bf837b1371afd06829e8e"
server_key_iv: "a45139abd0f8d027eb5034008c7917d2"
server_key_version: "encryptionkey@0"
I20260806 03:42:44.345474 7446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:44.358512 7446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:44.360836 7446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:44.360989 7446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:44.361231 7446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:44.361539 7446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:44.361595 7446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.361636 7446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:44.361670 7446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.367493 7446 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:44475
I20260806 03:42:44.367568 7577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:44475 every 8 connection(s)
I20260806 03:42:44.367930 7446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:44.370869 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7446
I20260806 03:42:44.371225 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:44921
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:40855
--webserver_port=34529
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.376603 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.368752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:56327)
0806 03:42:44.368939 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.368983 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.369504 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.369746 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.369750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.370325 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.370528 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.370533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.371678 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.371679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.372593 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.372596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.373140 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.373142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.373299 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.373979 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.373991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.376396 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
W20260806 03:42:44.376662 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.368771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56327 (local address 127.30.24.62:40855)
0806 03:42:44.368842 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.368844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.369067 (+ 223us) server_negotiation.cc:407] Connection header received
0806 03:42:44.369591 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.369593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.369606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.369669 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.370681 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.371572 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.372677 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.373029 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.373107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.376564 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:56327: BlockingRecv error: recv got EOF from 127.30.24.2:56327 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 03:42:44.376920 7578 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:40855 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.24.62:40855: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.382241 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.377203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:40595)
0806 03:42:44.377266 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.377278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.377323 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.377469 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.377470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.377582 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.377646 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.377647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.378712 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.378714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.379221 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.379223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.379775 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.379776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.379833 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.380365 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.380370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.382190 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 03:42:44.382402 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.377253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:40595 (local address 127.30.24.62:40855)
0806 03:42:44.377323 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.377325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.377360 (+ 35us) server_negotiation.cc:407] Connection header received
0806 03:42:44.377390 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.377391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.377403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.377466 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.377720 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.378593 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.379310 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.379681 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.379769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.382309 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:40595: BlockingRecv error: recv got EOF from 127.30.24.2:40595 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.387135 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.382551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:59409)
0806 03:42:44.382594 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.382605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.382644 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.382796 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.382796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.382901 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.382928 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.382929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.383962 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.383963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.384428 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.384429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.384953 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.384954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.384995 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.385451 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.385456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.387088 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 03:42:44.387261 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.382621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:59409 (local address 127.30.24.62:40855)
0806 03:42:44.382666 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.382669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.382678 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:44.382705 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.382706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.382718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.382792 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.383002 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.383875 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.384549 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.384879 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.384954 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.387195 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:59409: BlockingRecv error: recv got EOF from 127.30.24.2:59409 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260806 03:42:44.387533 7578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:42:44.461527 7581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:44.461735 7581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:44.461758 7581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:44.462864 7581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:44.462903 7581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:44.462927 7581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:42:44.464309 7581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/krb5kdc/oryx_127.30.24.3.keytab
--principal=oryx/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:44921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=34529
--tserver_master_addrs=127.30.24.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:44.464569 7581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:44.464771 7581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:44.470248 7590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:44.470279 7588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.470325 7581 server_base.cc:1034] running on GCE node
W20260806 03:42:44.470288 7587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:44.470695 7581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:44.470903 7581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:44.472064 7581 hybrid_clock.cc:648] HybridClock initialized: now 1785987764472047 us; error 52 us; skew 500 ppm
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987764, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:42:44.474790 7581 init.cc:377] Logged in from keytab as oryx/127.30.24.3@KRBTEST.COM (short username oryx)
I20260806 03:42:44.475310 7581 webserver.cc:460] Webserver started at http://127.30.24.3:34529/ using document root <none> and password file <none>
I20260806 03:42:44.475577 7581 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:44.475672 7581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:44.477123 7581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:42:44.477824 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.478015 7581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:42:44.478116 7581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "b7c4c220c77244eeb1be3008eff3703c"
format_stamp: "Formatted at 2026-08-06 03:42:43 on dist-test-slave-10pc"
server_key: "4888019e868196796572a99c7d4ef15c"
server_key_iv: "b2065f275b6bf272c9eaf4529cb51bb2"
server_key_version: "encryptionkey@0"
I20260806 03:42:44.478492 7581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:44.488849 7581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:44.491166 7581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:44.491334 7581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:44.491564 7581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:44.491876 7581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:44.491928 7581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.491969 7581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:44.491998 7581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:44.498039 7581 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:44921
I20260806 03:42:44.498090 7712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:44921 every 8 connection(s)
I20260806 03:42:44.498433 7581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 03:42:44.506841 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.499201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:39081 (local address 127.30.24.62:40855)
0806 03:42:44.499256 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.499259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.499494 (+ 235us) server_negotiation.cc:407] Connection header received
0806 03:42:44.500180 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.500181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.500195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.500258 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.501381 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.501927 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.503032 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.503259 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.503349 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.506761 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:39081: BlockingRecv error: recv got EOF from 127.30.24.3:39081 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:42:44.506870 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.499173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:39081)
0806 03:42:44.499366 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.499406 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.499963 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.500340 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.500344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.501045 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.501270 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.501275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.502049 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.502050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.502932 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.502934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.503360 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.503360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.503495 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.504008 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.504016 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.506650 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
W20260806 03:42:44.507107 7713 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:40855 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.24.62:40855: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
I20260806 03:42:44.507874 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7581
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.514192 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.507394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:56655)
0806 03:42:44.507433 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.507443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.507482 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.508806 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.508807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.508990 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.509055 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.509058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.510100 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.510101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.510816 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.510819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.511245 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.511246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.511319 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.511803 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.511809 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.514103 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260806 03:42:44.514238 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.508549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:56655 (local address 127.30.24.62:40855)
0806 03:42:44.508639 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.508641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.508651 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:44.508678 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.508679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.508690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.508753 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.509168 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.509985 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.510927 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.511155 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.511243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.514190 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:56655: BlockingRecv error: recv got EOF from 127.30.24.3:56655 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 03:42:44.517544 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.508587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.30.24.62:40855)
0806 03:42:44.508658 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.508660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.508669 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:44.508824 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.508825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.508862 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.508909 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.510042 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.510799 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.511638 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.511868 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.511956 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.513559 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:44.513565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:44.513565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:42:44.513580 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:44.515780 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:44.516192 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:44.516196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:44.516196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:44.516328 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:42:44.516616 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:42:44.516617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:42:44.516618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:42:44.516956 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:44.517152 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:44.517291 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 03:42:44.517395 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":38}
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:44 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:44.519857 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:44.514867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:41917)
0806 03:42:44.514979 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:44.514992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:44.515067 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:44.515194 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:44.515195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:44.515334 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:44.515366 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.515368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.516116 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.516117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.516993 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:44.516996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.517512 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.517513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:44.517558 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.517983 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:44.517988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:44.519777 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 03:42:44.519920 7445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:44.514929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:41917 (local address 127.30.24.62:40855)
0806 03:42:44.514977 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:44.514978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:44.515073 (+ 95us) server_negotiation.cc:407] Connection header received
0806 03:42:44.515131 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:44.515132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:44.515140 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:44.515187 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:44.515442 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.515980 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.517101 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:44.517389 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:44.517454 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:44.519847 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:41917: BlockingRecv error: recv got EOF from 127.30.24.3:41917 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":22}
W20260806 03:42:44.520206 7713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 03:42:45 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:45 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:45.261662 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:45.256666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:36557)
0806 03:42:45.256819 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:45.256832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:45.256914 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:45.257056 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:45.257057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:45.257225 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:45.257278 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:45.257279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.257912 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.257913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:45.258493 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:45.258495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.258853 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.258854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:45.258924 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:45.259385 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:45.259390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:45.261573 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
W20260806 03:42:45.261729 7719 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:45.256744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:36557 (local address 127.30.24.62:40855)
0806 03:42:45.256862 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:45.256864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:45.256890 (+ 26us) server_negotiation.cc:407] Connection header received
0806 03:42:45.256961 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:45.256961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:45.256971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:45.257022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:45.257339 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.257837 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.258580 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.258769 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.258821 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:45.261675 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:36557: BlockingRecv error: recv got EOF from 127.30.24.1:36557 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
Aug 06 03:42:45 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:45 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:45.393025 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:45.387940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:48281)
0806 03:42:45.388117 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:45.388130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:45.388225 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:45.388380 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:45.388382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:45.388615 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:45.388667 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:45.388670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.389454 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.389455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:45.390250 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:45.390253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.390620 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.390621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:45.390669 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:45.391165 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:45.391170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:45.392953 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
W20260806 03:42:45.393173 7719 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:45.388019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48281 (local address 127.30.24.62:40855)
0806 03:42:45.388100 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:45.388103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:45.388203 (+ 100us) server_negotiation.cc:407] Connection header received
0806 03:42:45.388286 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:45.388287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:45.388301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:45.388371 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:45.388769 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.389309 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.390342 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.390549 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.390615 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:45.393099 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:48281: BlockingRecv error: recv got EOF from 127.30.24.2:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 03:42:45 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:45 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:45.525336 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:45.520590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:38697)
0806 03:42:45.520734 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:45.520747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:45.520822 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:45.520943 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:45.520943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:45.521117 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:45.521199 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:45.521201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.521860 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.521861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:45.522476 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:45.522479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.522811 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.522812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:45.522872 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:45.523366 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:45.523371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:45.525263 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
W20260806 03:42:45.525406 7719 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:45.520679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:38697 (local address 127.30.24.62:40855)
0806 03:42:45.520732 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:45.520735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:45.520814 (+ 79us) server_negotiation.cc:407] Connection header received
0806 03:42:45.520868 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:45.520869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:45.520881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:45.520927 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:45.521257 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.521771 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.522555 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:45.522743 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:45.522792 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:45.525355 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:38697: BlockingRecv error: recv got EOF from 127.30.24.3:38697 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 06 03:42:46 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:46 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:46.266909 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:46.262162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:48043)
0806 03:42:46.262293 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:46.262305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:46.262362 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:46.262635 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:46.262637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:46.262799 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:46.262854 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:46.262855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.263505 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.263506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:46.264040 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:46.264042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.264392 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.264393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:46.264436 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:46.265016 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:46.265022 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:46.266839 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260806 03:42:46.267051 7723 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:46.262375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:48043 (local address 127.30.24.62:40855)
0806 03:42:46.262509 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:46.262511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:46.262519 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:46.262541 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:46.262542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:46.262552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:46.262616 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:46.262920 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.263436 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.264110 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.264325 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.264376 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:46.266940 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:48043: BlockingRecv error: recv got EOF from 127.30.24.1:48043 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Aug 06 03:42:46 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:46 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:46.398226 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:46.393499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:37349)
0806 03:42:46.393666 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:46.393679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:46.393749 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:46.393895 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:46.393896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:46.394096 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:46.394152 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:46.394154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.394842 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.394843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:46.395369 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:46.395370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.395733 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.395734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:46.395799 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:46.396298 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:46.396303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:46.398149 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20260806 03:42:46.398355 7723 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:46.393576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:37349 (local address 127.30.24.62:40855)
0806 03:42:46.393658 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:46.393660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:46.393728 (+ 68us) server_negotiation.cc:407] Connection header received
0806 03:42:46.393806 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:46.393807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:46.393819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:46.393886 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:46.394237 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.394758 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.395452 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.395649 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.395721 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:46.398256 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:37349: BlockingRecv error: recv got EOF from 127.30.24.2:37349 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 03:42:46 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:46 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:46.530500 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:46.525847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:49409)
0806 03:42:46.526022 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:46.526033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:46.526091 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:46.526218 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:46.526219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:46.526387 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:46.526440 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:46.526441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.527137 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.527138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:46.527655 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:46.527657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.528102 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.528103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:46.528149 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:46.528651 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:46.528655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:46.530437 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
W20260806 03:42:46.530691 7723 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:46.526001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:49409 (local address 127.30.24.62:40855)
0806 03:42:46.526072 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:46.526075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:46.526091 (+ 16us) server_negotiation.cc:407] Connection header received
0806 03:42:46.526138 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:46.526139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:46.526152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:46.526204 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:46.526502 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.527044 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.527745 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:46.528015 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:46.528080 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:46.530592 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:49409: BlockingRecv error: recv got EOF from 127.30.24.3:49409 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 03:42:47 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:47 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:47.272286 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:47.267434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:35511)
0806 03:42:47.267590 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:47.267605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:47.267673 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:47.267985 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:47.267986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:47.268205 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:47.268273 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:47.268276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.268980 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.268981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:47.269597 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:47.269599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.269972 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.269973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:47.270014 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:47.270467 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:47.270471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:47.272226 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260806 03:42:47.272406 7727 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:47.267522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:35511 (local address 127.30.24.62:40855)
0806 03:42:47.267664 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:47.267666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:47.267675 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:47.267871 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:47.267872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:47.267886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:47.267959 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:47.268350 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.268887 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.269692 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.269898 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.269963 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:47.272324 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:35511: BlockingRecv error: recv got EOF from 127.30.24.1:35511 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
Aug 06 03:42:47 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:47 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:47.403632 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:47.398713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:43857)
0806 03:42:47.398899 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:47.398916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:47.398992 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:47.399155 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:47.399156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:47.399342 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:47.399391 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:47.399393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.400076 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.400077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:47.400662 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:47.400664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.401050 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.401051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:47.401114 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:47.401562 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:47.401566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:47.403541 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
W20260806 03:42:47.403708 7727 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:47.398781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:43857 (local address 127.30.24.62:40855)
0806 03:42:47.398875 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:47.398878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:47.398970 (+ 92us) server_negotiation.cc:407] Connection header received
0806 03:42:47.399061 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:47.399062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:47.399076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:47.399138 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:47.399476 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.399991 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.400757 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.400961 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.401043 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:47.403646 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:43857: BlockingRecv error: recv got EOF from 127.30.24.2:43857 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 03:42:47 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:47 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:47.536209 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:47.530977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:41141)
0806 03:42:47.531186 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:47.531200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:47.531298 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:47.531484 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:47.531486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:47.531687 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:47.531741 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:47.531743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.532605 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.532607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:47.533206 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:47.533208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.533619 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.533620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:47.533694 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:47.534226 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:47.534231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:47.536104 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
W20260806 03:42:47.536264 7727 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:47.531099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:41141 (local address 127.30.24.62:40855)
0806 03:42:47.531188 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:47.531191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:47.531274 (+ 83us) server_negotiation.cc:407] Connection header received
0806 03:42:47.531394 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:47.531395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:47.531410 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:47.531473 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:47.531850 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.532495 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.533303 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:47.533519 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:47.533617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:47.536191 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:41141: BlockingRecv error: recv got EOF from 127.30.24.3:41141 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 03:42:48 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:48 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:48.281631 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:48.272751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:41855)
0806 03:42:48.272875 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:48.272888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:48.272962 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:48.273130 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:48.273131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:48.273309 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:48.273361 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:48.273363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.274101 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.274102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:48.274645 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:48.274647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.274970 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.274971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:48.275028 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:48.279560 (+ 4532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:48.279565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:48.281515 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
W20260806 03:42:48.281679 7731 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:48.272823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:41855 (local address 127.30.24.62:40855)
0806 03:42:48.272970 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:48.272971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:48.272982 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:42:48.273010 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:48.273010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:48.273021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:48.273117 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:48.273426 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.274008 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.274703 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.274904 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.274952 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:48.281607 (+ 6655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:41855: BlockingRecv error: recv got EOF from 127.30.24.1:41855 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
Aug 06 03:42:48 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:48 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:48.413476 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:48.404143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:46851)
0806 03:42:48.404317 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:48.404328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:48.404396 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:48.404585 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:48.404585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:48.404751 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:48.404803 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:48.404805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.405566 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.405568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:48.406108 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:48.406110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.406548 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.406549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:48.406602 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:48.411284 (+ 4682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:48.411289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:48.413415 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
W20260806 03:42:48.413607 7731 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:48.404240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:46851 (local address 127.30.24.62:40855)
0806 03:42:48.404312 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:48.404314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:48.404380 (+ 66us) server_negotiation.cc:407] Connection header received
0806 03:42:48.404494 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:48.404495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:48.404515 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:48.404564 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:48.404873 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.405456 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.406203 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.406474 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.406564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:48.413536 (+ 6972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:46851: BlockingRecv error: recv got EOF from 127.30.24.2:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 03:42:48 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:48 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:48.541474 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:48.536744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:52393)
0806 03:42:48.536884 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:48.536896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:48.536973 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:48.537112 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:48.537113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:48.537283 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:48.537334 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:48.537335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.537969 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.537970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:48.538498 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:48.538500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.538836 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.538837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:48.538877 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:48.539458 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:48.539463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:48.541372 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
W20260806 03:42:48.541549 7731 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:48.536830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52393 (local address 127.30.24.62:40855)
0806 03:42:48.536916 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:48.536918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:48.536955 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:42:48.537038 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:48.537039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:48.537054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:48.537107 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:48.537394 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.537897 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.538581 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:48.538769 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:48.538827 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:48.541458 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:52393: BlockingRecv error: recv got EOF from 127.30.24.3:52393 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 03:42:49 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:49 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:49.287163 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:49.282126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:41917)
0806 03:42:49.282269 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:49.282279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:49.282328 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:49.282462 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:49.282463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:49.282628 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:49.282680 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:49.282681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.283359 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.283360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:49.283900 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:49.283901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.284309 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.284309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:49.284354 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:49.284883 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:49.284888 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:49.287038 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260806 03:42:49.287214 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:49.282205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:41917 (local address 127.30.24.62:40855)
0806 03:42:49.282333 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:49.282335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:49.282344 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:49.282372 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:49.282373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:49.282384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:49.282450 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:49.282751 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.283287 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.283997 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.284228 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.284298 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:49.287130 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:41917: BlockingRecv error: recv got EOF from 127.30.24.1:41917 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
Aug 06 03:42:49 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:49 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:49.418864 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:49.414042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:59891)
0806 03:42:49.414184 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:49.414193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:49.414244 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:49.414413 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:49.414414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:49.414601 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:49.414654 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:49.414655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.415312 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.415313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:49.415872 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:49.415874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.416254 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.416259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:49.416311 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:49.416789 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:49.416794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:49.418750 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
W20260806 03:42:49.418939 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:49.414160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:59891 (local address 127.30.24.62:40855)
0806 03:42:49.414209 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:49.414211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:49.414229 (+ 18us) server_negotiation.cc:407] Connection header received
0806 03:42:49.414312 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:49.414313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:49.414324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:49.414377 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:49.414725 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.415240 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.415954 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.416158 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.416215 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:49.418863 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:59891: BlockingRecv error: recv got EOF from 127.30.24.2:59891 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 06 03:42:49 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:49 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:49.546581 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:49.541957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:52599)
0806 03:42:49.542116 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:49.542129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:49.542214 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:49.542360 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:49.542361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:49.542545 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:49.542598 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:49.542600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.543285 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.543286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:49.543809 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:49.543811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.544155 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.544156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:49.544218 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:49.544711 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:49.544716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:49.546481 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
W20260806 03:42:49.546657 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:49.542043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52599 (local address 127.30.24.62:40855)
0806 03:42:49.542117 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:49.542119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:49.542193 (+ 74us) server_negotiation.cc:407] Connection header received
0806 03:42:49.542271 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:49.542272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:49.542287 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:49.542352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:49.542678 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.543194 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.543889 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:49.544084 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:49.544155 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:49.546565 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:52599: BlockingRecv error: recv got EOF from 127.30.24.3:52599 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 06 03:42:50 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:50 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:50.292627 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:50.287690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:36069)
0806 03:42:50.287858 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:50.287869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:50.287932 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:50.288084 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:50.288085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:50.288263 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:50.288322 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:50.288323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.289075 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.289076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:50.289691 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:50.289692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.290080 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.290081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:50.290128 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:50.290699 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:50.290705 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:50.292562 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
W20260806 03:42:50.292804 7739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:50.287793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:36069 (local address 127.30.24.62:40855)
0806 03:42:50.287920 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:50.287922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:50.287931 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:50.287996 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:50.287997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:50.288009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:50.288061 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:50.288419 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.288993 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.289771 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.289966 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.290028 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:50.292710 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:36069: BlockingRecv error: recv got EOF from 127.30.24.1:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
Aug 06 03:42:50 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:50 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:50.424736 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:50.419538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:56979)
0806 03:42:50.419749 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:50.419762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:50.419861 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:50.420114 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:50.420115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:50.420347 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:50.420409 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:50.420411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.421211 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.421213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:50.421766 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:50.421768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.422119 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.422120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:50.422177 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:50.422707 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:50.422712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:50.424634 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
W20260806 03:42:50.424787 7739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:50.419639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56979 (local address 127.30.24.62:40855)
0806 03:42:50.419736 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:50.419739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:50.419838 (+ 99us) server_negotiation.cc:407] Connection header received
0806 03:42:50.419939 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:50.419941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:50.419959 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:50.420100 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:50.420580 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.421129 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.421852 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.422049 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.422123 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:50.424720 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:56979: BlockingRecv error: recv got EOF from 127.30.24.2:56979 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 03:42:50 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:50 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:50.551962 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:50.547113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:47443)
0806 03:42:50.547241 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:50.547258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:50.547313 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:50.547460 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:50.547461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:50.547641 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:50.547693 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:50.547695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.548367 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.548368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:50.549098 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:50.549102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.549429 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.549431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:50.549475 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:50.549944 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:50.549948 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:50.551864 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
W20260806 03:42:50.552063 7739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:50.547221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:47443 (local address 127.30.24.62:40855)
0806 03:42:50.547285 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:50.547287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:50.547301 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:42:50.547369 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:50.547370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:50.547383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:50.547444 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:50.547769 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.548293 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.549173 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:50.549361 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:50.549419 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:50.551986 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:47443: BlockingRecv error: recv got EOF from 127.30.24.3:47443 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 06 03:42:51 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:51 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:51.297926 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:51.293135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:50569)
0806 03:42:51.293269 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:51.293287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:51.293338 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:51.293501 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:51.293502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:51.293672 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:51.293726 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:51.293727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.294412 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.294413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:51.294945 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:51.294947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.295321 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.295321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:51.295367 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:51.295933 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:51.295937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:51.297864 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260806 03:42:51.298110 7743 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:51.293236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:50569 (local address 127.30.24.62:40855)
0806 03:42:51.293369 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:51.293371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:51.293379 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:42:51.293407 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:51.293411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:51.293424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:51.293488 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:51.293796 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.294328 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.295049 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.295246 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.295303 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:51.297997 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:50569: BlockingRecv error: recv got EOF from 127.30.24.1:50569 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
Aug 06 03:42:51 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:51 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:51.430828 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:51.425385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:56697)
0806 03:42:51.425555 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:51.425590 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:51.425692 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:51.425869 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:51.425871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:51.426107 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:51.426163 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:51.426166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.426878 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.426880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:51.427415 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:51.427416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.427824 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.427825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:51.427889 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:51.428518 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:51.428523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:51.430730 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
W20260806 03:42:51.430913 7743 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:51.425527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:56697 (local address 127.30.24.62:40855)
0806 03:42:51.425615 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:51.425618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:51.425663 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:42:51.425772 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:51.425773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:51.425785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:51.425857 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:51.426260 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.426788 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.427527 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.427735 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.427812 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:51.430845 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:56697: BlockingRecv error: recv got EOF from 127.30.24.2:56697 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 03:42:51 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:51 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:51.557693 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:51.552541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:45901)
0806 03:42:51.552727 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:51.552737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:51.552821 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:51.552983 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:51.552984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:51.553205 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:51.553259 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:51.553261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.553965 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.553967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:51.554499 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:51.554501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.554883 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.554884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:51.554958 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:51.555461 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:51.555466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:51.557549 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
W20260806 03:42:51.557832 7743 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:51.552586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:45901 (local address 127.30.24.62:40855)
0806 03:42:51.552673 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:51.552676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:51.552797 (+ 121us) server_negotiation.cc:407] Connection header received
0806 03:42:51.552889 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:51.552890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:51.552904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:51.552974 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:51.553345 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.553871 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.554586 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:51.554792 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:51.554879 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:51.557696 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:45901: BlockingRecv error: recv got EOF from 127.30.24.3:45901 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 03:42:52 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:52 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:52.303624 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:52.298544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:43009)
0806 03:42:52.298718 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:52.298734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:52.298813 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:52.299002 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:52.299004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:52.299190 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:52.299245 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:52.299247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.300032 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.300033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:52.300733 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:52.300737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.301067 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.301068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:52.301110 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:52.301686 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:52.301697 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:52.303562 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
W20260806 03:42:52.303758 7747 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:52.298706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:43009 (local address 127.30.24.62:40855)
0806 03:42:52.298836 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:52.298838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:52.298860 (+ 22us) server_negotiation.cc:407] Connection header received
0806 03:42:52.298885 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:52.298886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:52.298896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:52.298946 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:52.299319 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.299930 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.300805 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.301001 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.301055 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:52.303668 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:43009: BlockingRecv error: recv got EOF from 127.30.24.1:43009 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
Aug 06 03:42:52 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:52 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:52.436384 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:52.431348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:48105)
0806 03:42:52.431537 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:52.431551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:52.431637 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:52.431787 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:52.431788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:52.431979 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:52.432028 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:52.432030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.432936 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.432937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:52.433464 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:52.433466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.433839 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.433840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:52.433906 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:52.434378 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:52.434382 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:52.436282 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
W20260806 03:42:52.436439 7747 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:52.431439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48105 (local address 127.30.24.62:40855)
0806 03:42:52.431519 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:52.431521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:52.431617 (+ 96us) server_negotiation.cc:407] Connection header received
0806 03:42:52.431713 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:52.431714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:52.431729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:52.431783 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:52.432111 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.432850 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.433554 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.433762 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.433839 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:52.436373 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:48105: BlockingRecv error: recv got EOF from 127.30.24.2:48105 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 03:42:52 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:52 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:52.563621 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:52.558204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:40221)
0806 03:42:52.558418 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:52.558431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:52.558540 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:52.558732 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:52.558734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:52.558944 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:52.558996 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:52.558998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.559819 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.559820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:52.560528 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:52.560531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.561071 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.561073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:52.561148 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:52.561617 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:52.561623 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:52.563560 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":134,"threads_started":1}
W20260806 03:42:52.563818 7747 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:52.558320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:40221 (local address 127.30.24.62:40855)
0806 03:42:52.558411 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:52.558414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:52.558514 (+ 100us) server_negotiation.cc:407] Connection header received
0806 03:42:52.558625 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:52.558626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:52.558642 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:52.558725 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:52.559100 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.559712 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.560630 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:52.560959 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:52.561067 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:52.563695 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:40221: BlockingRecv error: recv got EOF from 127.30.24.3:40221 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 03:42:53 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:53 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:53.309698 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:53.304144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:38543)
0806 03:42:53.304319 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:53.304331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:53.304388 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:53.304571 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:53.304572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:53.304758 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:53.304813 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:53.304815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.305697 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.305699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:53.306408 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:53.306411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.306809 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.306810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:53.306860 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:53.307467 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:53.307472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:53.309585 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
W20260806 03:42:53.309765 7751 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:53.304237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:38543 (local address 127.30.24.62:40855)
0806 03:42:53.304354 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:53.304356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:53.304369 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:53.304474 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:53.304475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:53.304486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:53.304551 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:53.304911 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.305593 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.306518 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.306728 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.306790 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:53.309686 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:38543: BlockingRecv error: recv got EOF from 127.30.24.1:38543 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
Aug 06 03:42:53 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:53 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:53.446854 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:53.436972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:53877)
0806 03:42:53.437130 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:53.437148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:53.437208 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:53.437433 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:53.437434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:53.437631 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:53.437688 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:53.437690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.438406 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.438407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:53.438947 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:53.438949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.439314 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.439315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:53.439367 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:53.444400 (+ 5033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:53.444405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:53.446781 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
W20260806 03:42:53.447001 7751 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:53.437149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:53877 (local address 127.30.24.62:40855)
0806 03:42:53.437222 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:53.437223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:53.437256 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:42:53.437303 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:53.437304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:53.437317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:53.437361 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:53.437801 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.438312 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.439029 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.439224 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.439280 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:53.446915 (+ 7635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:53877: BlockingRecv error: recv got EOF from 127.30.24.2:53877 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 03:42:53 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:53 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:53.569162 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:53.564259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:32971)
0806 03:42:53.564484 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:53.564497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:53.564592 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:53.564771 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:53.564772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:53.564965 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:53.565027 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:53.565028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.565722 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.565723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:53.566320 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:53.566323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.566690 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.566691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:53.566756 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:53.567223 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:53.567227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:53.569066 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":115,"threads_started":1}
W20260806 03:42:53.569213 7751 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:53.564360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:32971 (local address 127.30.24.62:40855)
0806 03:42:53.564478 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:53.564481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:53.564569 (+ 88us) server_negotiation.cc:407] Connection header received
0806 03:42:53.564668 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:53.564669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:53.564689 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:53.564764 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:53.565118 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.565623 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.566417 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:53.566609 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:53.566687 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:53.569142 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:32971: BlockingRecv error: recv got EOF from 127.30.24.3:32971 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 03:42:54 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:54 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:54.315366 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:54.310266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:58053)
0806 03:42:54.310430 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:54.310442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:54.310495 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:54.310640 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:54.310641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:54.310823 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:54.310878 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:54.310880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.311586 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.311587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:54.312231 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:54.312233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.312609 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.312610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:54.312657 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:54.313158 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:54.313164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:54.315301 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
W20260806 03:42:54.315526 7755 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:54.310364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:58053 (local address 127.30.24.62:40855)
0806 03:42:54.310502 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:54.310504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:54.310513 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:54.310539 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:54.310540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:54.310550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:54.310618 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:54.310955 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.311479 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.312311 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.312527 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.312589 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:54.315430 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:58053: BlockingRecv error: recv got EOF from 127.30.24.1:58053 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
Aug 06 03:42:54 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:54 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:54.452661 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:54.447394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:36691)
0806 03:42:54.447597 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:54.447614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:54.447677 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:54.447832 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:54.447832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:54.448004 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:54.448065 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:54.448066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.448902 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.448903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:54.449516 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:54.449519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.449863 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.449864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:54.449907 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:54.450478 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:54.450482 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:54.452600 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260806 03:42:54.452785 7755 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:54.447460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:36691 (local address 127.30.24.62:40855)
0806 03:42:54.447534 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:54.447537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:54.447678 (+ 141us) server_negotiation.cc:407] Connection header received
0806 03:42:54.447731 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:54.447732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:54.447745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:54.447811 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:54.448146 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.448831 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.449604 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.449797 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.449851 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:54.452703 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:36691: BlockingRecv error: recv got EOF from 127.30.24.2:36691 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 03:42:54 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:54 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:54.574932 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:54.569705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:33795)
0806 03:42:54.569868 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:54.569885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:54.569966 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:54.570104 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:54.570105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:54.570293 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:54.570346 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:54.570348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.571238 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.571240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:54.571890 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:54.571894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.572243 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.572244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:54.572288 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:54.572869 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:54.572874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:54.574832 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260806 03:42:54.574996 7755 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:54.569869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:33795 (local address 127.30.24.62:40855)
0806 03:42:54.569951 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:54.569953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:54.569965 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:54.570038 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:54.570039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:54.570053 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:54.570100 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:54.570426 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.571128 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.571979 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:54.572171 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:54.572226 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:54.574911 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:33795: BlockingRecv error: recv got EOF from 127.30.24.3:33795 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 03:42:55 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:55 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:55.320695 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:55.315848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:47785)
0806 03:42:55.315989 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:55.316003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:55.316120 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:55.316298 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:55.316299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:55.316580 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:55.316647 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:55.316650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.317365 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.317366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:55.317910 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:55.317912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.318286 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.318286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:55.318331 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:55.318793 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:55.318797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:55.320631 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
W20260806 03:42:55.320834 7759 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:55.315954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:47785 (local address 127.30.24.62:40855)
0806 03:42:55.316115 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:55.316117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:55.316130 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:42:55.316180 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:55.316181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:55.316196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:55.316265 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:55.316740 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.317272 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.317995 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.318210 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.318271 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:55.320723 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:47785: BlockingRecv error: recv got EOF from 127.30.24.1:47785 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
Aug 06 03:42:55 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:55 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:55.458442 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:55.453221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:48935)
0806 03:42:55.453429 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:55.453442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:55.453561 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:55.453749 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:55.453750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:55.453950 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:55.454000 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:55.454002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.454693 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.454694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:55.455244 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:55.455246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.455623 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.455624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:55.455688 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:55.456243 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:55.456248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:55.458381 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
W20260806 03:42:55.458592 7759 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:55.453334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48935 (local address 127.30.24.62:40855)
0806 03:42:55.453420 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:55.453422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:55.453539 (+ 117us) server_negotiation.cc:407] Connection header received
0806 03:42:55.453649 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:55.453650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:55.453663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:55.453741 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:55.454095 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.454595 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.455338 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.455541 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.455624 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:55.458503 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:48935: BlockingRecv error: recv got EOF from 127.30.24.2:48935 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 03:42:55 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:55 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:55.580843 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:55.575565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:58601)
0806 03:42:55.575794 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:55.575809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:55.575921 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:55.576135 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:55.576136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:55.576350 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:55.576402 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:55.576404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.577234 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.577236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:55.577851 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:55.577853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.578261 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.578262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:55.578331 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:55.578814 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:55.578819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:55.580733 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":123,"threads_started":1}
W20260806 03:42:55.580905 7759 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:55.575656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:58601 (local address 127.30.24.62:40855)
0806 03:42:55.575774 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:55.575777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:55.575896 (+ 119us) server_negotiation.cc:407] Connection header received
0806 03:42:55.576011 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:55.576012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:55.576045 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:55.576123 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:55.576523 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.577124 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.577951 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:55.578174 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:55.578263 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:55.580832 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:58601: BlockingRecv error: recv got EOF from 127.30.24.3:58601 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 03:42:56 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:56 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:56.326084 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:56.321242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:51105)
0806 03:42:56.321407 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:56.321421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:56.321497 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:56.321650 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:56.321651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:56.321830 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:56.321883 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:56.321884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.322553 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.322553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:56.323076 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:56.323078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.323454 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.323455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:56.323500 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:56.323975 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:56.323980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:56.326005 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260806 03:42:56.326272 7763 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:56.321316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:51105 (local address 127.30.24.62:40855)
0806 03:42:56.321496 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:56.321498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:56.321507 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:56.321550 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:56.321551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:56.321567 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:56.321629 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:56.321960 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.322486 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.323185 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.323379 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.323431 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:56.326167 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:51105: BlockingRecv error: recv got EOF from 127.30.24.1:51105 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
Aug 06 03:42:56 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:56 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:56.463935 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:56.458941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:33451)
0806 03:42:56.459105 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:56.459121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:56.459180 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:56.459329 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:56.459330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:56.459506 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:56.459559 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:56.459561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.460214 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.460215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:56.460799 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:56.460801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.461314 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.461316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:56.461387 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:56.461856 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:56.461860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:56.463812 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
W20260806 03:42:56.463958 7763 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:56.459060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:33451 (local address 127.30.24.62:40855)
0806 03:42:56.459109 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:56.459111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:56.459174 (+ 63us) server_negotiation.cc:407] Connection header received
0806 03:42:56.459230 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:56.459231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:56.459243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:56.459297 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:56.459639 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.460139 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.460919 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.461229 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.461302 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:56.463893 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:33451: BlockingRecv error: recv got EOF from 127.30.24.2:33451 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 06 03:42:56 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:56 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:56.586153 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:56.581376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:38895)
0806 03:42:56.581532 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:56.581545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:56.581612 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:56.581755 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:56.581756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:56.581928 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:56.581979 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:56.581980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.582712 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.582713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:56.583278 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:56.583281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.583652 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.583653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:56.583698 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:56.584215 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:56.584220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:56.586068 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":110,"threads_started":1}
W20260806 03:42:56.586313 7763 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:56.581453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:38895 (local address 127.30.24.62:40855)
0806 03:42:56.581539 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:56.581541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:56.581599 (+ 58us) server_negotiation.cc:407] Connection header received
0806 03:42:56.581674 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:56.581675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:56.581690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:56.581739 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:56.582040 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.582610 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.583379 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:56.583578 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:56.583645 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:56.586228 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:38895: BlockingRecv error: recv got EOF from 127.30.24.3:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 03:42:57 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:57 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:57.331677 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:57.326676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:54081)
0806 03:42:57.326863 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:57.326878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:57.326965 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:57.327156 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:57.327157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:57.327343 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:57.327394 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:57.327395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.328072 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.328073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:57.328710 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:57.328712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.329069 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.329070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:57.329120 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:57.329588 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:57.329593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:57.331566 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
W20260806 03:42:57.331741 7767 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:57.326800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:54081 (local address 127.30.24.62:40855)
0806 03:42:57.326945 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:57.326947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:57.326957 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:42:57.327030 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:57.327031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:57.327043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:57.327126 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:57.327466 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.327973 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.328786 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.328989 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.329045 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:57.331658 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:54081: BlockingRecv error: recv got EOF from 127.30.24.1:54081 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
Aug 06 03:42:57 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:57 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:57.469069 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:57.464546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:39321)
0806 03:42:57.464669 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:57.464682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:57.464744 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:57.464916 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:57.464917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:57.465115 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:57.465167 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:57.465169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.465804 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.465804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:57.466333 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:57.466335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.466644 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.466645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:57.466685 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:57.467129 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:57.467133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:57.469009 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
W20260806 03:42:57.469281 7767 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:57.464713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:39321 (local address 127.30.24.62:40855)
0806 03:42:57.464783 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:57.464785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:57.464794 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:57.464825 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:57.464826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:57.464842 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:57.464893 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:57.465248 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.465739 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.466405 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.466584 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.466635 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:57.469138 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:39321: BlockingRecv error: recv got EOF from 127.30.24.2:39321 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 03:42:57 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:57 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:57.591491 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:57.586644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:52575)
0806 03:42:57.586806 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:57.586817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:57.586897 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:57.587074 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:57.587075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:57.587271 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:57.587321 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:57.587322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.588055 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.588056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:57.588631 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:57.588633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.589012 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.589013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:57.589075 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:57.589586 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:57.589591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:57.591388 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
W20260806 03:42:57.591531 7767 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:57.586698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52575 (local address 127.30.24.62:40855)
0806 03:42:57.586768 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:57.586771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:57.586875 (+ 104us) server_negotiation.cc:407] Connection header received
0806 03:42:57.586973 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:57.586974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:57.586989 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:57.587066 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:57.587425 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.587957 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.588722 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:57.588922 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:57.589009 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:57.591470 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:52575: BlockingRecv error: recv got EOF from 127.30.24.3:52575 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 03:42:58 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:58 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:58.337458 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:58.332311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:49693)
0806 03:42:58.332552 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:58.332565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:58.332653 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:58.332822 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:58.332823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:58.333019 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:58.333073 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:58.333075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.333802 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.333803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:58.334369 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:58.334370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.334842 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.334844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:58.334916 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:58.335541 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:58.335546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:58.337383 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
W20260806 03:42:58.337539 7771 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:58.332433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:49693 (local address 127.30.24.62:40855)
0806 03:42:58.332623 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:58.332625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:58.332637 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:42:58.332721 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:58.332722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:58.332735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:58.332797 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:58.333189 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.333701 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.334466 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.334737 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.334809 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:58.337483 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:49693: BlockingRecv error: recv got EOF from 127.30.24.1:49693 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":114,"threads_started":1}
Aug 06 03:42:58 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:58 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:58.479414 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:58.469653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:37203)
0806 03:42:58.469958 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:58.469970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:58.470061 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:58.470274 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:58.470275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:58.470467 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:58.470517 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:58.470519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.471212 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.471213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:58.471766 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:58.471768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.472178 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.472179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:58.472245 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:58.477409 (+ 5164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:58.477414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:58.479342 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
W20260806 03:42:58.479563 7771 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:58.469769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:37203 (local address 127.30.24.62:40855)
0806 03:42:58.469949 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:58.469951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:58.470041 (+ 90us) server_negotiation.cc:407] Connection header received
0806 03:42:58.470178 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:58.470179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:58.470192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:58.470266 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:58.470609 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.471119 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.471866 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.472093 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.472166 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:58.479479 (+ 7313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:37203: BlockingRecv error: recv got EOF from 127.30.24.2:37203 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Aug 06 03:42:58 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:58 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.3@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:58.598631 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:58.592036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.3:56123)
0806 03:42:58.592213 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:58.592228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:58.592314 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:58.592539 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:58.592540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:58.592789 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:58.592856 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:58.592859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.593881 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.593882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:58.594789 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:58.594792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.595332 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.595333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:58.595437 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:58.596014 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:58.596020 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:58.598519 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
W20260806 03:42:58.598848 7771 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:58.592181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:56123 (local address 127.30.24.62:40855)
0806 03:42:58.592233 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:58.592236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:58.592288 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:42:58.592395 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:58.592396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:58.592414 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:58.592525 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:58.592950 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.593785 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.594876 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:58.595219 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:58.595310 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:58.598641 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.3:56123: BlockingRecv error: recv got EOF from 127.30.24.3:56123 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 06 03:42:59 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:59 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.1@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:59.343402 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.338085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.1:57333)
0806 03:42:59.338240 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:59.338251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:59.338314 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:59.338511 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:59.338511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:59.338690 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:59.338749 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.338751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.339400 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.339400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.339929 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.339930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.340323 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.340324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.340367 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.340887 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:59.340892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:59.343330 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260806 03:42:59.343577 7775 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:59.338235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:57333 (local address 127.30.24.62:40855)
0806 03:42:59.338370 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:59.338371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:59.338380 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:42:59.338408 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:59.338409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:59.338420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:59.338493 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:59.338820 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.339330 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.340009 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.340250 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.340308 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.343459 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.1:57333: BlockingRecv error: recv got EOF from 127.30.24.1:57333 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
Aug 06 03:42:59 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:42:59 dist-test-slave-10pc krb5kdc[6690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.24.2@KRBTEST.COM for oryx/127.30.24.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:42:59.485024 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.479985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40855 (local address 127.30.24.2:52941)
0806 03:42:59.480172 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:59.480187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:59.480261 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:59.480410 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:59.480411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:59.480607 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:59.480672 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.480673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.481386 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.481388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.482008 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.482011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.482385 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.482386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.482445 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.482913 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:42:59.482918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:59.484950 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.24.62:40855: Server oryx/127.30.24.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
W20260806 03:42:59.485140 7775 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:42:59.480068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:52941 (local address 127.30.24.62:40855)
0806 03:42:59.480129 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:59.480131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:59.480258 (+ 127us) server_negotiation.cc:407] Connection header received
0806 03:42:59.480333 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:59.480334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:59.480345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:59.480389 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:59.480757 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.481276 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.482096 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.482296 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.482384 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.485052 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.24.2:52941: BlockingRecv error: recv got EOF from 127.30.24.2:52941 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:42:59.516973 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7290
I20260806 03:42:59.523696 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7446
I20260806 03:42:59.529438 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7581
I20260806 03:42:59.534365 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7219
2026-08-06T03:42:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17512 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T03:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:42:59Z Disabled control of system clock
I20260806 03:42:59.564868 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40019
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:40019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 03:42:59.647763 7780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:59.647979 7780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:59.648000 7780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:59.649230 7780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:42:59.649271 7780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:59.649284 7780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:59.649296 7780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:42:59.649309 7780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:42:59.650605 7780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:40019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:59.650815 7780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:59.651063 7780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:59.656280 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:59.656291 7788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:59.656394 7780 server_base.cc:1034] running on GCE node
W20260806 03:42:59.656534 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:59.656798 7780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:59.657091 7780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:59.658310 7780 hybrid_clock.cc:648] HybridClock initialized: now 1785987779658296 us; error 28 us; skew 500 ppm
I20260806 03:42:59.659579 7780 webserver.cc:460] Webserver started at http://127.30.24.62:40981/ using document root <none> and password file <none>
I20260806 03:42:59.659840 7780 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:59.659914 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:59.660065 7780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:59.661242 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "16091c17297c41818b7a48d98050975e"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "b944651d32427f4d44a847675744a917"
server_key_iv: "ac17e9abdf24f42ef0f0c20268354cdc"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.661624 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "16091c17297c41818b7a48d98050975e"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "b944651d32427f4d44a847675744a917"
server_key_iv: "ac17e9abdf24f42ef0f0c20268354cdc"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.662966 7780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:42:59.663810 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.664001 7780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:59.664134 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "16091c17297c41818b7a48d98050975e"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "b944651d32427f4d44a847675744a917"
server_key_iv: "ac17e9abdf24f42ef0f0c20268354cdc"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.664216 7780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:59.677423 7780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:59.677716 7780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:59.677870 7780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:59.681998 7780 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:40019
I20260806 03:42:59.682049 7850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:40019 every 8 connection(s)
I20260806 03:42:59.682394 7780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:42:59.683039 7851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:59.685787 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e: Bootstrap starting.
I20260806 03:42:59.686335 7851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:59.686569 7851 log.cc:824] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:42:59.687230 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e: No bootstrap required, opened a new log
I20260806 03:42:59.688714 7851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16091c17297c41818b7a48d98050975e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40019 } }
I20260806 03:42:59.688848 7851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:59.688869 7851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16091c17297c41818b7a48d98050975e, State: Initialized, Role: FOLLOWER
I20260806 03:42:59.689006 7851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16091c17297c41818b7a48d98050975e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40019 } }
I20260806 03:42:59.689100 7851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:59.689123 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:59.689157 7851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:59.689441 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7780
I20260806 03:42:59.689548 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:42:59.689675 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16091c17297c41818b7a48d98050975e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40019 } }
I20260806 03:42:59.689777 30816 external_mini_cluster.cc:1428] Setting key 936e4f37186855676e826d4d7d6e833d
I20260806 03:42:59.689782 7851 leader_election.cc:302] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16091c17297c41818b7a48d98050975e; no voters:
I20260806 03:42:59.689934 7851 leader_election.cc:288] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:59.689982 7854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:59.690198 7851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:59.690203 7854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [term 1 LEADER]: Becoming Leader. State: Replica: 16091c17297c41818b7a48d98050975e, State: Running, Role: LEADER
I20260806 03:42:59.690645 7854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16091c17297c41818b7a48d98050975e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40019 } }
I20260806 03:42:59.691287 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16091c17297c41818b7a48d98050975e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16091c17297c41818b7a48d98050975e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40019 } } }
I20260806 03:42:59.691371 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:59.691372 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16091c17297c41818b7a48d98050975e. Latest consensus state: current_term: 1 leader_uuid: "16091c17297c41818b7a48d98050975e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16091c17297c41818b7a48d98050975e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 40019 } } }
I20260806 03:42:59.691697 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e [sys.catalog]: This master's current role is: LEADER
W20260806 03:42:59.692302 7872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:42:59.692360 7872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:42:59.692425 7868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:59.692632 7868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:59.696014 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.690388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.30.24.62:40019)
0806 03:42:59.690657 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:59.690667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:59.690700 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:42:59.690824 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:59.690832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:59.690901 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:59.691320 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:59.691448 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.692852 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.693892 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.694491 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.694635 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.695296 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:59.695300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:59.695300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:59.695516 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:59.695558 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:59.695573 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:59.695654 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:59.695712 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 03:42:59.695792 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260806 03:42:59.696291 7868 catalog_manager.cc:1347] Generated new cluster ID: 81eed5e030d64b039aec8570dbe0711e
I20260806 03:42:59.696343 7868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:59.711772 7868 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:59.712528 7868 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:59.721938 7868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16091c17297c41818b7a48d98050975e: Generated new TSK 0
I20260806 03:42:59.722142 7868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:59.724109 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40019
--builtin_ntp_servers=127.30.24.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 03:42:59.808315 7876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:59.808591 7876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:59.808647 7876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:59.809880 7876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:59.809942 7876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:59.809986 7876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:42:59.811362 7876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:59.811632 7876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:59.811949 7876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:59.817916 7884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:59.817927 7881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:59.817927 7882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:59.818228 7876 server_base.cc:1034] running on GCE node
I20260806 03:42:59.818466 7876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:59.818677 7876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:59.819844 7876 hybrid_clock.cc:648] HybridClock initialized: now 1785987779819833 us; error 32 us; skew 500 ppm
I20260806 03:42:59.821238 7876 webserver.cc:460] Webserver started at http://127.30.24.1:45539/ using document root <none> and password file <none>
I20260806 03:42:59.821501 7876 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:59.821574 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:59.821745 7876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:59.822832 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "40a85c6d9ef6469e818cd88637d7e0f3"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "83abaceebb0a44a0a80b938524366712"
server_key_iv: "1551b3c624e5dced9c73552cb383a5f3"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.823200 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "40a85c6d9ef6469e818cd88637d7e0f3"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "83abaceebb0a44a0a80b938524366712"
server_key_iv: "1551b3c624e5dced9c73552cb383a5f3"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.824564 7876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:59.825707 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.825886 7876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:42:59.825979 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "40a85c6d9ef6469e818cd88637d7e0f3"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "83abaceebb0a44a0a80b938524366712"
server_key_iv: "1551b3c624e5dced9c73552cb383a5f3"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.826066 7876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:59.838001 7876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:59.838321 7876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:59.838470 7876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:59.838690 7876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:59.839012 7876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:59.839063 7876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.839097 7876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:59.839162 7876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.845036 7876 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:34919
I20260806 03:42:59.845096 8006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:34919 every 8 connection(s)
I20260806 03:42:59.845413 7876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:42:59.849886 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 7876
I20260806 03:42:59.850001 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:42:59.850235 30816 external_mini_cluster.cc:1428] Setting key a98186c491206e8a8221b9af0e1c4d38
I20260806 03:42:59.850741 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.846147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:50061 (local address 127.30.24.62:40019)
0806 03:42:59.846215 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:59.846218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:59.846466 (+ 248us) server_negotiation.cc:407] Connection header received
0806 03:42:59.846959 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:59.846960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:59.846973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:59.847026 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:59.847660 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.848152 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.849164 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.849349 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.849400 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.850329 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:59.850331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:59.850331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:59.850339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:59.850357 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:59.850362 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:59.850415 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:59.850625 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 03:42:59.850663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:42:59.850951 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.846145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40019 (local address 127.30.24.1:50061)
0806 03:42:59.846345 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:59.846381 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:59.846873 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:59.847080 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:59.847083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:59.847092 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:59.847587 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.847591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.848235 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.848236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.849082 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.849085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.849425 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.849426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.849534 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.850156 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:59.850156 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:59.850199 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:59.850228 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:59.850522 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:59.850528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:59.850531 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:42:59.850609 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 03:42:59.850719 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260806 03:42:59.851305 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40019
--builtin_ntp_servers=127.30.24.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 03:42:59.851460 8007 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40019
I20260806 03:42:59.851554 8007 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:59.851748 8007 heartbeater.cc:511] Master 127.30.24.62:40019 requested a full tablet report, sending...
I20260806 03:42:59.852268 7815 ts_manager.cc:194] Registered new tserver with Master: 40a85c6d9ef6469e818cd88637d7e0f3 (127.30.24.1:34919)
I20260806 03:42:59.852918 7815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.1:50061
W20260806 03:42:59.938360 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:42:59.938560 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:42:59.938581 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:42:59.939651 8011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:42:59.939683 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:42:59.939708 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:42:59.940977 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:42:59.941174 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:42:59.941418 8011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:59.946328 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:59.946429 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:59.946566 8011 server_base.cc:1034] running on GCE node
W20260806 03:42:59.946426 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:59.946830 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:42:59.947072 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:42:59.948242 8011 hybrid_clock.cc:648] HybridClock initialized: now 1785987779948240 us; error 35 us; skew 500 ppm
I20260806 03:42:59.949470 8011 webserver.cc:460] Webserver started at http://127.30.24.2:42169/ using document root <none> and password file <none>
I20260806 03:42:59.949730 8011 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:59.949801 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:59.949946 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:59.951010 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "a4df14fb4a874524961c08ba9b02fb5a"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "5c224efab1b7a9e7ea120dd1baedff3d"
server_key_iv: "83526f13b29d5e58b0defc9d17004968"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.951399 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "a4df14fb4a874524961c08ba9b02fb5a"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "5c224efab1b7a9e7ea120dd1baedff3d"
server_key_iv: "83526f13b29d5e58b0defc9d17004968"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.952766 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:42:59.953492 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.953651 8011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:42:59.953752 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "a4df14fb4a874524961c08ba9b02fb5a"
format_stamp: "Formatted at 2026-08-06 03:42:59 on dist-test-slave-10pc"
server_key: "5c224efab1b7a9e7ea120dd1baedff3d"
server_key_iv: "83526f13b29d5e58b0defc9d17004968"
server_key_version: "encryptionkey@0"
I20260806 03:42:59.953835 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:59.975404 8011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:59.975759 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:42:59.975930 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:59.976188 8011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:59.976599 8011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:59.976658 8011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.976701 8011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:59.976728 8011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:59.982726 8011 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:44631
I20260806 03:42:59.982781 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:44631 every 8 connection(s)
I20260806 03:42:59.983106 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:42:59.987134 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8011
I20260806 03:42:59.987257 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:42:59.987452 30816 external_mini_cluster.cc:1428] Setting key 760864d09b9d83cdc03827fb90c7d517
I20260806 03:42:59.988548 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40019
--builtin_ntp_servers=127.30.24.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 03:42:59.988548 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.983883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:41973 (local address 127.30.24.62:40019)
0806 03:42:59.983932 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:42:59.983934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:42:59.984184 (+ 250us) server_negotiation.cc:407] Connection header received
0806 03:42:59.984702 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:42:59.984703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:42:59.984713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:42:59.984784 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:42:59.985449 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.985956 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.986942 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.987145 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.987199 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.988243 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:42:59.988244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:42:59.988245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:42:59.988249 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:42:59.988263 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:42:59.988269 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:42:59.988319 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 03:42:59.988418 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 03:42:59.988469 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:42:59.988775 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:42:59.983844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40019 (local address 127.30.24.2:41973)
0806 03:42:59.984045 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 03:42:59.984095 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:42:59.984612 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:42:59.984841 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:42:59.984844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:42:59.984853 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:42:59.985369 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.985373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.986041 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.986041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.986843 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:42:59.986845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:42:59.987237 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:42:59.987238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:42:59.987415 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:42:59.988067 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:42:59.988067 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:42:59.988097 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:42:59.988137 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:42:59.988325 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:42:59.988330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:42:59.988334 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:42:59.988420 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 03:42:59.988545 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260806 03:42:59.989280 8143 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40019
I20260806 03:42:59.989400 8143 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:59.989619 8143 heartbeater.cc:511] Master 127.30.24.62:40019 requested a full tablet report, sending...
I20260806 03:42:59.990065 7815 ts_manager.cc:194] Registered new tserver with Master: a4df14fb4a874524961c08ba9b02fb5a (127.30.24.2:44631)
I20260806 03:42:59.990540 7815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.2:41973
W20260806 03:43:00.072901 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:00.073108 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:00.073129 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:00.074210 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:00.074244 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:00.074267 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:43:00.075508 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:00.075697 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:00.075942 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:00.080899 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.080956 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.080906 8155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.081115 8146 server_base.cc:1034] running on GCE node
I20260806 03:43:00.081321 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:00.081570 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:00.082724 8146 hybrid_clock.cc:648] HybridClock initialized: now 1785987780082714 us; error 28 us; skew 500 ppm
I20260806 03:43:00.083958 8146 webserver.cc:460] Webserver started at http://127.30.24.3:41293/ using document root <none> and password file <none>
I20260806 03:43:00.084196 8146 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:00.084242 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:00.084388 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:00.085493 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "65d036063a2146cb8757be7aec03de64"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "61ac74fcfe9bf94e273cf16ca74315a7"
server_key_iv: "48bf7c4c8c3d04f2c27f11da6f2e84fd"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.085873 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "65d036063a2146cb8757be7aec03de64"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "61ac74fcfe9bf94e273cf16ca74315a7"
server_key_iv: "48bf7c4c8c3d04f2c27f11da6f2e84fd"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.087224 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:43:00.087927 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.088097 8146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:43:00.088188 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "65d036063a2146cb8757be7aec03de64"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "61ac74fcfe9bf94e273cf16ca74315a7"
server_key_iv: "48bf7c4c8c3d04f2c27f11da6f2e84fd"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.088263 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:00.095537 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:00.095817 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:00.095961 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:00.096177 8146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:00.096551 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:00.096588 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.096611 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:00.096652 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.103282 8146 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:43645
I20260806 03:43:00.103410 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:43645 every 8 connection(s)
I20260806 03:43:00.103703 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:43:00.104583 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8146
I20260806 03:43:00.104687 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:43:00.104940 30816 external_mini_cluster.cc:1428] Setting key 4b865ed6d4b1d3640d16db468d693f8d
I20260806 03:43:00.109215 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.104649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:39397 (local address 127.30.24.62:40019)
0806 03:43:00.104706 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.104708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.104858 (+ 150us) server_negotiation.cc:407] Connection header received
0806 03:43:00.105483 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.105484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.105495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.105538 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.106221 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:00.106739 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:00.107792 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:00.107994 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:00.108073 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:00.108934 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:00.108936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:00.108936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:43:00.108941 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:00.108955 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:43:00.108959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:00.109014 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:00.109102 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 03:43:00.109135 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:43:00.109438 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.104492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:40019 (local address 127.30.24.3:39397)
0806 03:43:00.104712 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:00.104752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:00.105397 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:00.105607 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:00.105611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:00.105621 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:00.106126 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:00.106131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:00.106839 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:00.106840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:00.107684 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:00.107687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:00.108082 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:00.108084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:00.108230 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:00.108795 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:43:00.108796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:00.108824 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:43:00.108847 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:00.109017 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:00.109022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:00.109025 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:43:00.109105 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:43:00.109217 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260806 03:43:00.109858 8277 heartbeater.cc:347] Connected to a master server at 127.30.24.62:40019
I20260806 03:43:00.109965 8277 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:00.110167 8277 heartbeater.cc:511] Master 127.30.24.62:40019 requested a full tablet report, sending...
I20260806 03:43:00.110538 7815 ts_manager.cc:194] Registered new tserver with Master: 65d036063a2146cb8757be7aec03de64 (127.30.24.3:43645)
I20260806 03:43:00.110865 7815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.3:39397
I20260806 03:43:00.116830 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 03:43:00.118619 7858 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:43:00.118150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.30.24.62:40019)
0806 03:43:00.118225 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.118226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.118318 (+ 92us) server_negotiation.cc:407] Connection header received
0806 03:43:00.118356 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.118356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.118367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.118417 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.118536 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35506: BlockingRecv error: recv got EOF from 127.0.0.1:35506 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 03:43:00.118727 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7876
I20260806 03:43:00.125784 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8011
I20260806 03:43:00.131668 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8146
I20260806 03:43:00.136551 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 7780
2026-08-06T03:43:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (600 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T03:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:43:00Z Disabled control of system clock
I20260806 03:43:00.164810 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42095
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:42095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 03:43:00.252724 8289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:00.252929 8289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:00.252951 8289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:00.254117 8289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:43:00.254158 8289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:00.254171 8289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:00.254184 8289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:43:00.254236 8289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:43:00.255617 8289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:42095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:00.255874 8289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:00.256199 8289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:00.261369 8298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.261343 8294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.261351 8295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.261445 8289 server_base.cc:1034] running on GCE node
I20260806 03:43:00.261772 8289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:00.262075 8289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:00.263263 8289 hybrid_clock.cc:648] HybridClock initialized: now 1785987780263256 us; error 38 us; skew 500 ppm
I20260806 03:43:00.264567 8289 webserver.cc:460] Webserver started at http://127.30.24.62:36313/ using document root <none> and password file <none>
I20260806 03:43:00.264835 8289 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:00.264907 8289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:00.265075 8289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:00.266172 8289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "2b97194f297b4b0f8aa0809902097d3b"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "23154d2fb386a60645ece2e068d3ecdd"
server_key_iv: "68aec7bd0b43b1538ea255ad081320d6"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.266548 8289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "2b97194f297b4b0f8aa0809902097d3b"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "23154d2fb386a60645ece2e068d3ecdd"
server_key_iv: "68aec7bd0b43b1538ea255ad081320d6"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.267871 8289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:43:00.268646 8304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.268819 8289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:43:00.268944 8289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "2b97194f297b4b0f8aa0809902097d3b"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "23154d2fb386a60645ece2e068d3ecdd"
server_key_iv: "68aec7bd0b43b1538ea255ad081320d6"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.269021 8289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:00.272343 8289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:00.272652 8289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:00.272804 8289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:00.276582 8289 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:42095
I20260806 03:43:00.276616 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:42095 every 8 connection(s)
I20260806 03:43:00.276954 8289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:43:00.277667 8358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:00.279286 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8289
I20260806 03:43:00.279397 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:43:00.279616 30816 external_mini_cluster.cc:1428] Setting key 093f670599ac8c2c6fc6c8ca42f9c6f7
I20260806 03:43:00.280969 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b: Bootstrap starting.
I20260806 03:43:00.281715 8358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:00.282100 8358 log.cc:824] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:00.282316 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.280198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.30.24.62:42095)
0806 03:43:00.280402 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.280409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.280434 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:43:00.280886 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.280893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.280934 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.281090 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.281799 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:00.281803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:00.281803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:43:00.281808 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:00.281858 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:43:00.281875 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:00.281949 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:00.282011 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 03:43:00.282078 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260806 03:43:00.282858 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b: No bootstrap required, opened a new log
I20260806 03:43:00.284227 8358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97194f297b4b0f8aa0809902097d3b" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42095 } }
I20260806 03:43:00.284387 8358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:00.284411 8358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b97194f297b4b0f8aa0809902097d3b, State: Initialized, Role: FOLLOWER
I20260806 03:43:00.284596 8358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97194f297b4b0f8aa0809902097d3b" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42095 } }
I20260806 03:43:00.284682 8358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:43:00.284724 8358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:43:00.284773 8358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:00.285631 8358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97194f297b4b0f8aa0809902097d3b" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42095 } }
I20260806 03:43:00.285723 8358 leader_election.cc:302] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b97194f297b4b0f8aa0809902097d3b; no voters:
I20260806 03:43:00.285924 8358 leader_election.cc:288] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:43:00.285974 8363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:43:00.286136 8363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [term 1 LEADER]: Becoming Leader. State: Replica: 2b97194f297b4b0f8aa0809902097d3b, State: Running, Role: LEADER
I20260806 03:43:00.286244 8358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:43:00.286238 8363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97194f297b4b0f8aa0809902097d3b" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42095 } }
I20260806 03:43:00.286645 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b97194f297b4b0f8aa0809902097d3b. Latest consensus state: current_term: 1 leader_uuid: "2b97194f297b4b0f8aa0809902097d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97194f297b4b0f8aa0809902097d3b" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42095 } } }
I20260806 03:43:00.286657 8364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b97194f297b4b0f8aa0809902097d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97194f297b4b0f8aa0809902097d3b" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42095 } } }
I20260806 03:43:00.286871 8364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [sys.catalog]: This master's current role is: LEADER
I20260806 03:43:00.286854 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b [sys.catalog]: This master's current role is: LEADER
I20260806 03:43:00.287240 8372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:43:00.287743 8372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:43:00.289258 8372 catalog_manager.cc:1347] Generated new cluster ID: 87193d16f7ad40bf933b587154e18011
I20260806 03:43:00.289312 8372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:43:00.298657 8372 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:43:00.298835 8372 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:43:00.305471 8372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b97194f297b4b0f8aa0809902097d3b: Generated new TSK 0
I20260806 03:43:00.305697 8372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:43:00.313182 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42095
--builtin_ntp_servers=127.30.24.20:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 03:43:00.403470 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:00.403679 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:00.403702 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:00.404893 8382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:00.404933 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:00.404958 8382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:43:00.406313 8382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:00.406562 8382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:00.406865 8382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:00.411960 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.411976 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.412201 8390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.412544 8382 server_base.cc:1034] running on GCE node
I20260806 03:43:00.412744 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:00.413023 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:00.414255 8382 hybrid_clock.cc:648] HybridClock initialized: now 1785987780414234 us; error 36 us; skew 500 ppm
I20260806 03:43:00.415583 8382 webserver.cc:460] Webserver started at http://127.30.24.1:34435/ using document root <none> and password file <none>
I20260806 03:43:00.415858 8382 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:00.415935 8382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:00.416093 8382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:00.417299 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "a09e038eef2a4e1aae939d4daf2fd9ce"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "08b4dd39258ef00603f0fa9b960209ad"
server_key_iv: "51b9ae9cfc41faee010c943c78c46337"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.417692 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "a09e038eef2a4e1aae939d4daf2fd9ce"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "08b4dd39258ef00603f0fa9b960209ad"
server_key_iv: "51b9ae9cfc41faee010c943c78c46337"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.419186 8382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:43:00.419998 8397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.420200 8382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:43:00.420300 8382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "a09e038eef2a4e1aae939d4daf2fd9ce"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "08b4dd39258ef00603f0fa9b960209ad"
server_key_iv: "51b9ae9cfc41faee010c943c78c46337"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.420392 8382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:00.423750 8382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:00.424026 8382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:00.424180 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:00.424402 8382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:00.424739 8382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:00.424794 8382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.424834 8382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:00.424860 8382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.430707 8382 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:34575
I20260806 03:43:00.430748 8512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:34575 every 8 connection(s)
I20260806 03:43:00.431114 8382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:43:00.434409 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.431910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:48403 (local address 127.30.24.62:42095)
0806 03:43:00.432110 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.432113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.432318 (+ 205us) server_negotiation.cc:407] Connection header received
0806 03:43:00.432900 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.432901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.432908 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.432966 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.434153 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:00.434155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:00.434155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:43:00.434160 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:00.434177 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:43:00.434182 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:00.434241 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:00.434320 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 03:43:00.434334 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 03:43:00.434608 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.431874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42095 (local address 127.30.24.1:48403)
0806 03:43:00.432193 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:00.432226 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:00.432790 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:00.433039 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:00.433043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:00.433053 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:00.433961 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:43:00.433961 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:00.433988 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:43:00.434017 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:00.434241 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:00.434248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:00.434251 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:43:00.434331 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:43:00.434394 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260806 03:43:00.435117 8513 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42095
I20260806 03:43:00.435218 8513 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:00.435370 8513 heartbeater.cc:511] Master 127.30.24.62:42095 requested a full tablet report, sending...
I20260806 03:43:00.435907 8321 ts_manager.cc:194] Registered new tserver with Master: a09e038eef2a4e1aae939d4daf2fd9ce (127.30.24.1:34575)
I20260806 03:43:00.439553 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8382
I20260806 03:43:00.439638 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:43:00.439791 30816 external_mini_cluster.cc:1428] Setting key 229ef7130fa4da2c29dad0b1bc282387
I20260806 03:43:00.440788 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42095
--builtin_ntp_servers=127.30.24.20:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 03:43:00.525390 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:00.525626 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:00.525666 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:00.526841 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:00.526902 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:00.526947 8516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:43:00.528306 8516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:00.528584 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:00.528908 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:00.534070 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.534183 8516 server_base.cc:1034] running on GCE node
W20260806 03:43:00.534040 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.534050 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.534535 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:00.534755 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:00.535897 8516 hybrid_clock.cc:648] HybridClock initialized: now 1785987780535883 us; error 28 us; skew 500 ppm
I20260806 03:43:00.537142 8516 webserver.cc:460] Webserver started at http://127.30.24.2:35269/ using document root <none> and password file <none>
I20260806 03:43:00.537379 8516 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:00.537448 8516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:00.537614 8516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:00.538717 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "2c7e12226c68405caee1c988024b0966"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "60e19236425d2bd66433cdc4b7572777"
server_key_iv: "199a4b280d0770e8acd3853fcc15666f"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.539098 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "2c7e12226c68405caee1c988024b0966"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "60e19236425d2bd66433cdc4b7572777"
server_key_iv: "199a4b280d0770e8acd3853fcc15666f"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.540418 8516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:43:00.541229 8531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.541422 8516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:43:00.541519 8516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "2c7e12226c68405caee1c988024b0966"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "60e19236425d2bd66433cdc4b7572777"
server_key_iv: "199a4b280d0770e8acd3853fcc15666f"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.541613 8516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:00.544958 8516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:00.545228 8516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:00.545372 8516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:00.545586 8516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:00.545885 8516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:00.545941 8516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.545979 8516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:00.546006 8516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.551756 8516 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:34975
I20260806 03:43:00.551800 8647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:34975 every 8 connection(s)
I20260806 03:43:00.552134 8516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:43:00.555428 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.552933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48323 (local address 127.30.24.62:42095)
0806 03:43:00.552996 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.552998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.553296 (+ 298us) server_negotiation.cc:407] Connection header received
0806 03:43:00.553832 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.553834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.553840 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.553909 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.555206 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:00.555208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:00.555208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:43:00.555213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:00.555229 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:43:00.555233 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:00.555294 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:00.555359 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 03:43:00.555372 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 03:43:00.555599 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.552872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42095 (local address 127.30.24.2:48323)
0806 03:43:00.553131 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:00.553194 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:00.553697 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:00.553989 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:00.553992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:00.554000 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:00.555041 (+ 1041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:43:00.555041 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:00.555070 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:43:00.555100 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:00.555294 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:00.555297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:00.555299 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:43:00.555350 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 03:43:00.555398 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":160,"threads_started":1}
I20260806 03:43:00.556102 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8516
I20260806 03:43:00.556242 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:43:00.556301 8648 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42095
I20260806 03:43:00.556406 8648 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:00.556512 30816 external_mini_cluster.cc:1428] Setting key 4acbb81c687701fc4e19e7ee9d7d0d5d
I20260806 03:43:00.556592 8648 heartbeater.cc:511] Master 127.30.24.62:42095 requested a full tablet report, sending...
I20260806 03:43:00.557037 8320 ts_manager.cc:194] Registered new tserver with Master: 2c7e12226c68405caee1c988024b0966 (127.30.24.2:34975)
I20260806 03:43:00.557597 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42095
--builtin_ntp_servers=127.30.24.20:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 03:43:00.641717 8651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:00.641920 8651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:00.641942 8651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:00.643095 8651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:00.643136 8651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:00.643170 8651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:43:00.644618 8651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:00.644860 8651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:00.645207 8651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:00.650372 8657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.650472 8651 server_base.cc:1034] running on GCE node
W20260806 03:43:00.650426 8659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:00.650395 8656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:00.650794 8651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:00.650993 8651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:00.652130 8651 hybrid_clock.cc:648] HybridClock initialized: now 1785987780652119 us; error 33 us; skew 500 ppm
I20260806 03:43:00.653348 8651 webserver.cc:460] Webserver started at http://127.30.24.3:42909/ using document root <none> and password file <none>
I20260806 03:43:00.653563 8651 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:00.653604 8651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:00.653713 8651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:00.654733 8651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "e92bfbd8bc9042548f3d2db00cb89533"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "737a415ce9425c0d55f31519ce6e58cf"
server_key_iv: "62fc31a29ee647d60df8e10a1f6b0dba"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.655061 8651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "e92bfbd8bc9042548f3d2db00cb89533"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "737a415ce9425c0d55f31519ce6e58cf"
server_key_iv: "62fc31a29ee647d60df8e10a1f6b0dba"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.656356 8651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:43:00.657155 8665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.657334 8651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:43:00.657408 8651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "e92bfbd8bc9042548f3d2db00cb89533"
format_stamp: "Formatted at 2026-08-06 03:43:00 on dist-test-slave-10pc"
server_key: "737a415ce9425c0d55f31519ce6e58cf"
server_key_iv: "62fc31a29ee647d60df8e10a1f6b0dba"
server_key_version: "encryptionkey@0"
I20260806 03:43:00.657462 8651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:00.660794 8651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:00.661017 8651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:00.661123 8651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:00.661280 8651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:00.661517 8651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:00.661547 8651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.661567 8651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:00.661586 8651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:00.667116 8651 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:38155
I20260806 03:43:00.667261 8779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:38155 every 8 connection(s)
I20260806 03:43:00.667474 8651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:43:00.670830 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.668241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:60537 (local address 127.30.24.62:42095)
0806 03:43:00.668291 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.668294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.668505 (+ 211us) server_negotiation.cc:407] Connection header received
0806 03:43:00.669042 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.669043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.669050 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.669116 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.670571 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:00.670573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:00.670573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:43:00.670578 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:00.670594 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:43:00.670598 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:00.670663 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:00.670744 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 03:43:00.670759 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:43:00.671097 8782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:00.668186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42095 (local address 127.30.24.3:60537)
0806 03:43:00.668359 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:00.668400 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:00.668928 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:00.669187 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:00.669190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:00.669200 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:00.670380 (+ 1180us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:43:00.670381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:00.670410 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:43:00.670438 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:00.670676 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:00.670681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:00.670684 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:43:00.670757 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 03:43:00.670818 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260806 03:43:00.671644 8780 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42095
I20260806 03:43:00.671728 8780 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:00.671895 8780 heartbeater.cc:511] Master 127.30.24.62:42095 requested a full tablet report, sending...
I20260806 03:43:00.672309 8320 ts_manager.cc:194] Registered new tserver with Master: e92bfbd8bc9042548f3d2db00cb89533 (127.30.24.3:38155)
I20260806 03:43:00.673426 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8651
I20260806 03:43:00.673511 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:43:00.673671 30816 external_mini_cluster.cc:1428] Setting key 59506b76c36876277fd93f33e44472e5
I20260806 03:43:00.674551 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 03:43:00.676229 8361 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:43:00.675762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.30.24.62:42095)
0806 03:43:00.675801 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:00.675804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:00.675950 (+ 146us) server_negotiation.cc:407] Connection header received
0806 03:43:00.675988 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:00.675989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:00.675993 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:00.676040 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:00.676165 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50306: BlockingRecv error: recv got EOF from 127.0.0.1:50306 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:43:00.676283 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8382
I20260806 03:43:00.683491 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8516
I20260806 03:43:00.688602 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8651
I20260806 03:43:00.693440 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8289
2026-08-06T03:43:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (556 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:43:00 dist-test-slave-10pc krb5kdc[8790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:43:00 dist-test-slave-10pc krb5kdc[8790](info): set up 2 sockets
Aug 06 03:43:00 dist-test-slave-10pc krb5kdc[8790](info): commencing operation
krb5kdc: starting...
W20260806 03:43:01.755208 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:43:01 dist-test-slave-10pc krb5kdc[8790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:43:01.881472 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35397
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:01.965058 8803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:01.965306 8803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:01.965343 8803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:01.966526 8803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:43:01.966594 8803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:01.966619 8803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:01.966640 8803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:43:01.966668 8803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:43:01.968189 8803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:35397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:01.968503 8803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:01.968751 8803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:43:01.973932 8803 server_base.cc:1034] running on GCE node
W20260806 03:43:01.973842 8811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:01.973821 8809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:01.973820 8808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:01.974388 8803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:01.974723 8803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:01.975900 8803 hybrid_clock.cc:648] HybridClock initialized: now 1785987781975878 us; error 35 us; skew 500 ppm
Aug 06 03:43:01 dist-test-slave-10pc krb5kdc[8790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987781, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:01.978399 8803 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:43:01.978850 8803 webserver.cc:460] Webserver started at http://127.30.24.62:36477/ using document root <none> and password file <none>
I20260806 03:43:01.979058 8803 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:01.979096 8803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:01.979194 8803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:01.980216 8803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3"
format_stamp: "Formatted at 2026-08-06 03:43:01 on dist-test-slave-10pc"
server_key: "2a7bb24161d727a25fbb8ac4f96a71ce"
server_key_iv: "8331e56de9493a4968b41b7dc1ed493e"
server_key_version: "encryptionkey@0"
I20260806 03:43:01.980587 8803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3"
format_stamp: "Formatted at 2026-08-06 03:43:01 on dist-test-slave-10pc"
server_key: "2a7bb24161d727a25fbb8ac4f96a71ce"
server_key_iv: "8331e56de9493a4968b41b7dc1ed493e"
server_key_version: "encryptionkey@0"
I20260806 03:43:01.981901 8803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:43:01.982653 8818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:01.982859 8803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:43:01.982949 8803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3"
format_stamp: "Formatted at 2026-08-06 03:43:01 on dist-test-slave-10pc"
server_key: "2a7bb24161d727a25fbb8ac4f96a71ce"
server_key_iv: "8331e56de9493a4968b41b7dc1ed493e"
server_key_version: "encryptionkey@0"
I20260806 03:43:01.983006 8803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:01.992702 8803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:01.995004 8803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:01.995122 8803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:01.998942 8803 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:35397
I20260806 03:43:01.998996 8871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:35397 every 8 connection(s)
I20260806 03:43:01.999336 8803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:43:02.000370 8872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:02.002725 8872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3: Bootstrap starting.
I20260806 03:43:02.003343 8872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:02.003650 8872 log.cc:824] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:02.004384 8872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3: No bootstrap required, opened a new log
I20260806 03:43:02.006080 8872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35397 } }
I20260806 03:43:02.006256 8872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:02.006301 8872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df6d5e1fc344bd380f1aaa5762a9dd3, State: Initialized, Role: FOLLOWER
I20260806 03:43:02.006510 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8803
I20260806 03:43:02.006501 8872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35397 } }
I20260806 03:43:02.006588 8872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:43:02.006628 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:43:02.006636 8872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:43:02.006686 8872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:02.006856 30816 external_mini_cluster.cc:1428] Setting key 0051986b4bfd0d887591a0eed3405be4
I20260806 03:43:02.007746 8872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35397 } }
I20260806 03:43:02.007877 8872 leader_election.cc:302] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df6d5e1fc344bd380f1aaa5762a9dd3; no voters:
I20260806 03:43:02.008087 8872 leader_election.cc:288] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:43:02.008128 8877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:43:02.008314 8877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 4df6d5e1fc344bd380f1aaa5762a9dd3, State: Running, Role: LEADER
I20260806 03:43:02.008379 8872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:43:02.008443 8877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35397 } }
I20260806 03:43:02.008919 8878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35397 } } }
I20260806 03:43:02.009002 8878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:43:02.009268 8879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df6d5e1fc344bd380f1aaa5762a9dd3. Latest consensus state: current_term: 1 leader_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df6d5e1fc344bd380f1aaa5762a9dd3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35397 } } }
I20260806 03:43:02.009423 8884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:43:02.009618 8879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:43:02.010241 8884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:43:02.011904 8884 catalog_manager.cc:1347] Generated new cluster ID: 66388341c1d6432aa3475c14c5fb7318
I20260806 03:43:02.011958 8884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:02.018911 8884 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:43:02.019452 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.007404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.30.24.62:35397)
0806 03:43:02.007616 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.007627 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.007660 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:43:02.007770 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.007776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.007818 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.007967 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:02.008862 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.010360 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.011317 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.011903 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.012016 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.015138 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:02.015142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:02.015143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:02.015155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:02.017294 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.017785 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.017788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.017789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.017910 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.018133 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.018134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.018135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.018440 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:02.018601 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.019067 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.019236 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260806 03:43:02.019549 8884 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:43:02.023808 8884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3: Generated new TSK 0
I20260806 03:43:02.023980 8884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:43:02.056236 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35397
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:02.140779 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:02.141000 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:02.141024 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:02.142198 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:02.142242 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:02.142268 8900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:43:02.143662 8900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:02.143847 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:02.144032 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:02.149093 8908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:02.149178 8905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:02.149260 8900 server_base.cc:1034] running on GCE node
W20260806 03:43:02.149189 8906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:02.149567 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:02.149819 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:02.151010 8900 hybrid_clock.cc:648] HybridClock initialized: now 1785987782150986 us; error 39 us; skew 500 ppm
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:02.153832 8900 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:43:02.154310 8900 webserver.cc:460] Webserver started at http://127.30.24.1:35865/ using document root <none> and password file <none>
I20260806 03:43:02.154539 8900 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:02.154614 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:02.154765 8900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:02.155828 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "96b706fb5f2a411d8e3da5c06de965f2"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "4a768bc02941e0c38f203a74f4941d5c"
server_key_iv: "f1d04bd40624b9d94fdd89d981779572"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.156198 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "96b706fb5f2a411d8e3da5c06de965f2"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "4a768bc02941e0c38f203a74f4941d5c"
server_key_iv: "f1d04bd40624b9d94fdd89d981779572"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.157632 8900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:43:02.158361 8915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.158537 8900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:43:02.158651 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "96b706fb5f2a411d8e3da5c06de965f2"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "4a768bc02941e0c38f203a74f4941d5c"
server_key_iv: "f1d04bd40624b9d94fdd89d981779572"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.158736 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:02.189020 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:02.191499 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:02.191684 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:02.191952 8900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:02.192287 8900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:02.192322 8900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.192382 8900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:02.192409 8900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.198513 8900 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:40723
I20260806 03:43:02.198571 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:40723 every 8 connection(s)
I20260806 03:43:02.198892 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:43:02.202018 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 8900
I20260806 03:43:02.202122 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:43:02.202335 30816 external_mini_cluster.cc:1428] Setting key 605ca1ea036bcae9a50a105edebe3776
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:02.210179 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.199681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:53367 (local address 127.30.24.62:35397)
0806 03:43:02.199773 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.199775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.199999 (+ 224us) server_negotiation.cc:407] Connection header received
0806 03:43:02.200605 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.200607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.200623 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.200686 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:02.201777 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.202517 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.203772 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.204022 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.204139 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.206798 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:02.206799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:02.206800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:02.206811 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:02.208504 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.208992 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.208994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.208994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.209069 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.209396 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.209397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.209397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.209544 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:02.209638 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.209969 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.210074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:43:02.210393 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.199649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35397 (local address 127.30.24.1:53367)
0806 03:43:02.199857 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.199906 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.200483 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.200780 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.200785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.201433 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:02.201662 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.201667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.202618 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.202619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.203668 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.203671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.204151 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.204152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.204297 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.204939 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:43:02.204952 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:02.206686 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:02.208621 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:02.208630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:02.208632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:02.208884 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:02.209155 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:02.209156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:02.209157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:02.209334 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:02.209654 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:02.209661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:02.209812 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.209930 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.210122 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260806 03:43:02.210978 9031 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35397
I20260806 03:43:02.211095 9031 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:02.211321 9031 heartbeater.cc:511] Master 127.30.24.62:35397 requested a full tablet report, sending...
I20260806 03:43:02.211963 8835 ts_manager.cc:194] Registered new tserver with Master: 96b706fb5f2a411d8e3da5c06de965f2 (127.30.24.1:40723)
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
I20260806 03:43:02.212772 8835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:53367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:43:02.230969 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35397
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:02.327034 9037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:02.327258 9037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:02.327286 9037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:02.328522 9037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:02.328583 9037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:02.328613 9037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:43:02.330135 9037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:02.330334 9037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:02.330530 9037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:02.335709 9042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:02.335773 9045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:02.335716 9043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:02.336107 9037 server_base.cc:1034] running on GCE node
I20260806 03:43:02.336333 9037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:02.336629 9037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:02.337812 9037 hybrid_clock.cc:648] HybridClock initialized: now 1785987782337808 us; error 41 us; skew 500 ppm
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:02.340389 9037 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:43:02.340940 9037 webserver.cc:460] Webserver started at http://127.30.24.2:34801/ using document root <none> and password file <none>
I20260806 03:43:02.341185 9037 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:02.341233 9037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:02.341393 9037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:02.342538 9037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "269339bafccf423f8c61271c681b26e8"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "a141a6926ee2403b61ec0f69e7cc324e"
server_key_iv: "db979b4d2fefeeeba22c09e93c1d3de1"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.342931 9037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "269339bafccf423f8c61271c681b26e8"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "a141a6926ee2403b61ec0f69e7cc324e"
server_key_iv: "db979b4d2fefeeeba22c09e93c1d3de1"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.344388 9037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:43:02.345314 9052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.345518 9037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:43:02.345633 9037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "269339bafccf423f8c61271c681b26e8"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "a141a6926ee2403b61ec0f69e7cc324e"
server_key_iv: "db979b4d2fefeeeba22c09e93c1d3de1"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.345731 9037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:02.386691 9037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:02.388803 9037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:02.388983 9037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:02.389254 9037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:02.389596 9037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:02.389653 9037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.389689 9037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:02.389717 9037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.395444 9037 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:39833
I20260806 03:43:02.395491 9166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:39833 every 8 connection(s)
I20260806 03:43:02.395828 9037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:43:02.397379 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 9037
I20260806 03:43:02.397524 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:43:02.397738 30816 external_mini_cluster.cc:1428] Setting key 8b6b8cb844c86a114bc62543cde61864
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:02.407042 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.396697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:53989 (local address 127.30.24.62:35397)
0806 03:43:02.396787 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.396790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.397001 (+ 211us) server_negotiation.cc:407] Connection header received
0806 03:43:02.397601 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.397602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.397617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.397673 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:02.398925 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.399438 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.400813 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.401014 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.401079 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.403729 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:02.403731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:02.403732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:02.403742 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:02.405468 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.405970 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.405972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.405972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.406043 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.406326 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.406327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.406327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.406455 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:02.406535 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.406838 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.406930 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
I20260806 03:43:02.407240 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.396665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35397 (local address 127.30.24.2:53989)
0806 03:43:02.396857 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.396906 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.397452 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.397762 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.397767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.398553 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:02.398809 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.398815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.399539 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.399541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.400722 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.400726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.401108 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.401109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.401259 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.401826 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:43:02.401837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:02.403603 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:02.405630 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:02.405638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:02.405640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:02.405882 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:02.406129 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:02.406130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:02.406130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:02.406246 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:02.406533 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:02.406541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:02.406694 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.406805 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.406982 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260806 03:43:02.407793 9167 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35397
I20260806 03:43:02.407913 9167 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:02.408140 9167 heartbeater.cc:511] Master 127.30.24.62:35397 requested a full tablet report, sending...
I20260806 03:43:02.408650 8835 ts_manager.cc:194] Registered new tserver with Master: 269339bafccf423f8c61271c681b26e8 (127.30.24.2:39833)
I20260806 03:43:02.409078 8835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:53989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:43:02.428570 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35397
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:02.515327 9172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:02.515542 9172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:02.515566 9172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:02.516785 9172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:02.516836 9172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:02.516865 9172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:43:02.518361 9172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:02.518637 9172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:02.518882 9172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:43:02.524248 9172 server_base.cc:1034] running on GCE node
W20260806 03:43:02.524207 9177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:02.524366 9178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:02.524199 9180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:02.524722 9172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:02.524940 9172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:02.526080 9172 hybrid_clock.cc:648] HybridClock initialized: now 1785987782526070 us; error 31 us; skew 500 ppm
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:02.528733 9172 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:43:02.529137 9172 webserver.cc:460] Webserver started at http://127.30.24.3:44385/ using document root <none> and password file <none>
I20260806 03:43:02.529340 9172 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:02.529381 9172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:02.529474 9172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:43:02.530500 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "fafbc67f030a40d3b9069e832e92aba2"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "a7f373c36962b9bf04ebc31c17f7a00c"
server_key_iv: "8609c65c4a792d684fe329b1351593ce"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.530814 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "fafbc67f030a40d3b9069e832e92aba2"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "a7f373c36962b9bf04ebc31c17f7a00c"
server_key_iv: "8609c65c4a792d684fe329b1351593ce"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.532181 9172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:43:02.533052 9187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.533262 9172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:43:02.533362 9172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "fafbc67f030a40d3b9069e832e92aba2"
format_stamp: "Formatted at 2026-08-06 03:43:02 on dist-test-slave-10pc"
server_key: "a7f373c36962b9bf04ebc31c17f7a00c"
server_key_iv: "8609c65c4a792d684fe329b1351593ce"
server_key_version: "encryptionkey@0"
I20260806 03:43:02.533452 9172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:02.548590 9172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:02.551088 9172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:02.551266 9172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:02.551587 9172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:02.552037 9172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:02.552083 9172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.552127 9172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:02.552162 9172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:02.558207 9172 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:34873
I20260806 03:43:02.558290 9304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:34873 every 8 connection(s)
I20260806 03:43:02.558636 9172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:43:02.564399 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 9172
I20260806 03:43:02.564555 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:43:02.564798 30816 external_mini_cluster.cc:1428] Setting key 8dd959e9434893952ec1e9363ddd8a26
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[8790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:02.568346 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.559417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:35817 (local address 127.30.24.62:35397)
0806 03:43:02.559506 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.559508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.559696 (+ 188us) server_negotiation.cc:407] Connection header received
0806 03:43:02.560207 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.560209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.560220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.560275 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:02.561414 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.561918 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.563066 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.563267 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.563325 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.565915 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:02.565916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:02.565917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:02.565923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:02.567087 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.567478 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.567478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.567479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.567532 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.567765 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.567765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.567766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.567875 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:02.567927 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.568205 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.568275 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 03:43:02.568593 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.559371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35397 (local address 127.30.24.3:35817)
0806 03:43:02.559557 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.559596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.560109 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.560333 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.560336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.561041 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:02.561329 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.561334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.562009 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.562010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.562990 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.562993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.563346 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.563347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.563458 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.563973 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:43:02.563982 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:02.565792 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:02.567173 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:02.567180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:02.567182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:02.567407 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:02.567591 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:02.567592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:02.567592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:02.567708 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:02.567931 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:02.567935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:02.568057 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.568172 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.568303 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260806 03:43:02.569118 9305 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35397
I20260806 03:43:02.569226 9305 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:02.569412 9305 heartbeater.cc:511] Master 127.30.24.62:35397 requested a full tablet report, sending...
I20260806 03:43:02.569808 8835 ts_manager.cc:194] Registered new tserver with Master: fafbc67f030a40d3b9069e832e92aba2 (127.30.24.3:34873)
I20260806 03:43:02.570207 8835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:35817
I20260806 03:43:02.576786 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:43:02.583938 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.578160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.30.24.62:35397)
0806 03:43:02.578205 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.578207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.578341 (+ 134us) server_negotiation.cc:407] Connection header received
0806 03:43:02.578372 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.578373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.578383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.578436 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:02.578958 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.579429 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.580207 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.580382 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.580432 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.581666 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:02.581667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:02.581667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:02.581673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:02.582918 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.583245 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.583246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.583246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.583296 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:02.583462 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:02.583462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:02.583462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:02.583619 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:02.583694 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.583783 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.583844 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:43:02.585714 8835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:43:02.587409 8835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:43:02.594607 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.590019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:40723 (local address 127.0.0.1:49504)
0806 03:43:02.590394 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.590409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.590475 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.590995 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.590996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.590999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:02.591053 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.591054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.592342 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.592343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.593391 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.593394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.594284 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.594292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.594365 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.594377 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.594455 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.594496 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260806 03:43:02.594635 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.590119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.30.24.1:40723)
0806 03:43:02.590636 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.590640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.590650 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:43:02.590711 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.590713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.590730 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.590821 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:02.591122 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.592233 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.593483 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.594150 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.594227 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.594288 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.594489 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.594539 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":90,"threads_started":1}
I20260806 03:43:02.594898 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.590009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:39833 (local address 127.0.0.1:33688)
0806 03:43:02.590336 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.590376 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.590444 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.591172 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.591173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.591175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:02.591208 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.591210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.592151 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.592152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.593705 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.593709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.594630 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.594631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.594702 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.594708 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.594775 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.594825 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":120,"thread_start_us":59,"threads_started":1}
I20260806 03:43:02.595549 8964 tablet_service.cc:1467] Processing CreateTablet for tablet 3379bb4c17534e949a1583b36a7d6077 (DEFAULT_TABLE table=test-table [id=d35815616a5a46c38c8e1be709f4f42c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:43:02.595748 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.590223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:34873 (local address 127.0.0.1:44254)
0806 03:43:02.590566 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.590575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.590611 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.590811 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.590816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.590818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:02.590897 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.590899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.592301 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.592302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.593722 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.593724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.595500 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.595502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.595576 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.595580 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.595637 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.595679 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":72,"thread_start_us":85,"threads_started":1}
I20260806 03:43:02.595768 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.590356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.30.24.3:34873)
0806 03:43:02.590598 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.590603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.590615 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:43:02.590697 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.590700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.590715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.590792 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:02.590972 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.592187 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.594697 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.595410 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.595497 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.595550 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.595653 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.595693 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":113,"threads_started":1}
I20260806 03:43:02.596076 8964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3379bb4c17534e949a1583b36a7d6077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:02.596144 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.590183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.30.24.2:39833)
0806 03:43:02.590978 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.590983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.590993 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:43:02.591035 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.591037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.591053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.591170 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:02.591276 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.592037 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.593812 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.594520 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.594601 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.594655 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.595699 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.596040 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":107,"threads_started":1}
I20260806 03:43:02.596483 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 3379bb4c17534e949a1583b36a7d6077 (DEFAULT_TABLE table=test-table [id=d35815616a5a46c38c8e1be709f4f42c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:43:02.596832 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3379bb4c17534e949a1583b36a7d6077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:02.596829 9101 tablet_service.cc:1467] Processing CreateTablet for tablet 3379bb4c17534e949a1583b36a7d6077 (DEFAULT_TABLE table=test-table [id=d35815616a5a46c38c8e1be709f4f42c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:43:02.597152 9101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3379bb4c17534e949a1583b36a7d6077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:02.599015 9322 tablet_bootstrap.cc:492] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: Bootstrap starting.
I20260806 03:43:02.599675 9322 tablet_bootstrap.cc:654] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:02.600167 9323 tablet_bootstrap.cc:492] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Bootstrap starting.
I20260806 03:43:02.600262 9322 log.cc:824] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:02.600345 9324 tablet_bootstrap.cc:492] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Bootstrap starting.
I20260806 03:43:02.600931 9323 tablet_bootstrap.cc:654] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:02.601068 9324 tablet_bootstrap.cc:654] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:02.601379 9322 tablet_bootstrap.cc:492] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: No bootstrap required, opened a new log
I20260806 03:43:02.601454 9322 ts_tablet_manager.cc:1397] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:43:02.601475 9323 log.cc:824] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:02.601625 9324 log.cc:824] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:02.602263 9323 tablet_bootstrap.cc:492] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: No bootstrap required, opened a new log
I20260806 03:43:02.602344 9323 ts_tablet_manager.cc:1397] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:43:02.602406 9324 tablet_bootstrap.cc:492] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: No bootstrap required, opened a new log
I20260806 03:43:02.602492 9324 ts_tablet_manager.cc:1397] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:43:02.603530 9322 raft_consensus.cc:348] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.603729 9322 raft_consensus.cc:374] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:02.603766 9322 raft_consensus.cc:729] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b706fb5f2a411d8e3da5c06de965f2, State: Initialized, Role: FOLLOWER
I20260806 03:43:02.603958 9322 consensus_queue.cc:260] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.604228 9322 ts_tablet_manager.cc:1428] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:43:02.604389 9323 raft_consensus.cc:348] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.604560 9323 raft_consensus.cc:374] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:02.604616 9323 raft_consensus.cc:729] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafbc67f030a40d3b9069e832e92aba2, State: Initialized, Role: FOLLOWER
I20260806 03:43:02.604729 9031 heartbeater.cc:503] Master 127.30.24.62:35397 was elected leader, sending a full tablet report...
I20260806 03:43:02.604615 9324 raft_consensus.cc:348] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.604739 9323 consensus_queue.cc:260] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.604796 9324 raft_consensus.cc:374] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:02.604831 9324 raft_consensus.cc:729] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269339bafccf423f8c61271c681b26e8, State: Initialized, Role: FOLLOWER
I20260806 03:43:02.604955 9324 consensus_queue.cc:260] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.605082 9323 ts_tablet_manager.cc:1428] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 03:43:02.605072 9329 raft_consensus.cc:482] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:02.605147 9329 raft_consensus.cc:504] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.605278 9324 ts_tablet_manager.cc:1428] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:43:02.605427 9167 heartbeater.cc:503] Master 127.30.24.62:35397 was elected leader, sending a full tablet report...
I20260806 03:43:02.605450 9305 heartbeater.cc:503] Master 127.30.24.62:35397 was elected leader, sending a full tablet report...
I20260806 03:43:02.606277 9329 leader_election.cc:288] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269339bafccf423f8c61271c681b26e8 (127.30.24.2:39833), fafbc67f030a40d3b9069e832e92aba2 (127.30.24.3:34873)
I20260806 03:43:02.609366 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.605879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:49905 (local address 127.30.24.2:39833)
0806 03:43:02.605940 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.605943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.605951 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:43:02.605976 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.605977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.605988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.606041 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:02.606333 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.607087 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.608343 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.608967 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.609079 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.609090 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.609263 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.609298 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 03:43:02.609368 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.605641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:39833 (local address 127.30.24.1:49905)
0806 03:43:02.605740 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.605753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.605794 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.606091 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.606092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.606094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:02.606158 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.606160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.607167 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.607168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.608256 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.608260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.609080 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.609081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.609191 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.609196 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.609254 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.609298 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 03:43:02.609658 9121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3379bb4c17534e949a1583b36a7d6077" candidate_uuid: "96b706fb5f2a411d8e3da5c06de965f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269339bafccf423f8c61271c681b26e8" is_pre_election: true
I20260806 03:43:02.609827 9121 raft_consensus.cc:2454] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b706fb5f2a411d8e3da5c06de965f2 in term 0.
I20260806 03:43:02.609882 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.606258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:43503 (local address 127.30.24.3:34873)
0806 03:43:02.606300 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.606302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.606449 (+ 147us) server_negotiation.cc:407] Connection header received
0806 03:43:02.606536 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.606537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.606549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.606623 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:02.606784 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.607532 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.608790 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.609385 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.609492 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.609505 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.609775 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.609818 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:43:02.609910 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.606069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:34873 (local address 127.30.24.1:43503)
0806 03:43:02.606394 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.606403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:02.606474 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:02.606627 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:02.606627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:02.606629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:02.606694 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.606696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.607625 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.607626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.608688 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:02.608691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.609597 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.609599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:02.609702 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.609707 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:43:02.609767 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 03:43:02.609814 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":168,"threads_started":1}
I20260806 03:43:02.610122 8919 leader_election.cc:302] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269339bafccf423f8c61271c681b26e8, 96b706fb5f2a411d8e3da5c06de965f2; no voters:
I20260806 03:43:02.610275 9329 raft_consensus.cc:2781] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:43:02.610248 9258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3379bb4c17534e949a1583b36a7d6077" candidate_uuid: "96b706fb5f2a411d8e3da5c06de965f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fafbc67f030a40d3b9069e832e92aba2" is_pre_election: true
I20260806 03:43:02.610330 9329 raft_consensus.cc:482] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:02.610351 9329 raft_consensus.cc:3037] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:02.610400 9258 raft_consensus.cc:2454] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b706fb5f2a411d8e3da5c06de965f2 in term 0.
I20260806 03:43:02.611106 9329 raft_consensus.cc:504] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.611243 9329 leader_election.cc:288] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [CANDIDATE]: Term 1 election: Requested vote from peers 269339bafccf423f8c61271c681b26e8 (127.30.24.2:39833), fafbc67f030a40d3b9069e832e92aba2 (127.30.24.3:34873)
I20260806 03:43:02.611436 9121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3379bb4c17534e949a1583b36a7d6077" candidate_uuid: "96b706fb5f2a411d8e3da5c06de965f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269339bafccf423f8c61271c681b26e8"
I20260806 03:43:02.611510 9121 raft_consensus.cc:3037] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:02.611521 9258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3379bb4c17534e949a1583b36a7d6077" candidate_uuid: "96b706fb5f2a411d8e3da5c06de965f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fafbc67f030a40d3b9069e832e92aba2"
I20260806 03:43:02.611620 9258 raft_consensus.cc:3037] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:02.612176 9121 raft_consensus.cc:2454] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b706fb5f2a411d8e3da5c06de965f2 in term 1.
I20260806 03:43:02.612363 8919 leader_election.cc:302] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269339bafccf423f8c61271c681b26e8, 96b706fb5f2a411d8e3da5c06de965f2; no voters:
I20260806 03:43:02.612509 9329 raft_consensus.cc:2781] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:43:02.612584 9329 raft_consensus.cc:686] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 1 LEADER]: Becoming Leader. State: Replica: 96b706fb5f2a411d8e3da5c06de965f2, State: Running, Role: LEADER
I20260806 03:43:02.612680 9329 consensus_queue.cc:237] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } }
I20260806 03:43:02.612882 9258 raft_consensus.cc:2454] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b706fb5f2a411d8e3da5c06de965f2 in term 1.
I20260806 03:43:02.613508 8835 catalog_manager.cc:5495] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b706fb5f2a411d8e3da5c06de965f2 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "96b706fb5f2a411d8e3da5c06de965f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b706fb5f2a411d8e3da5c06de965f2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:02.624629 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.622395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.30.24.1:40723)
0806 03:43:02.622433 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:02.622435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:02.622444 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:43:02.622504 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:02.622505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:02.622521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:02.622572 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:43:02.622712 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.623185 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.623907 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:02.624081 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:02.624149 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:02.624285 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:43:02.624383 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:43:02.624442 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:02.624535 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 03:43:02.624551 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:43:02.626458 9121 raft_consensus.cc:1261] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 1 FOLLOWER]: Refusing update from remote peer 96b706fb5f2a411d8e3da5c06de965f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:02.626478 9258 raft_consensus.cc:1261] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 1 FOLLOWER]: Refusing update from remote peer 96b706fb5f2a411d8e3da5c06de965f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:02.626847 9328 consensus_queue.cc:1035] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269339bafccf423f8c61271c681b26e8" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:43:02.626986 9329 consensus_queue.cc:1035] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fafbc67f030a40d3b9069e832e92aba2" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:43:02.632126 8834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:43:02.632239 8834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:43:02.633069 8834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4df6d5e1fc344bd380f1aaa5762a9dd3: Sending DeleteTablet for 3 replicas of tablet 3379bb4c17534e949a1583b36a7d6077
I20260806 03:43:02.633420 8964 tablet_service.cc:1514] Processing DeleteTablet for tablet 3379bb4c17534e949a1583b36a7d6077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:43:02 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:49504
I20260806 03:43:02.633507 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 3379bb4c17534e949a1583b36a7d6077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:43:02 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:44254
I20260806 03:43:02.633652 9346 tablet_replica.cc:330] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2: stopping tablet replica
I20260806 03:43:02.633657 9101 tablet_service.cc:1514] Processing DeleteTablet for tablet 3379bb4c17534e949a1583b36a7d6077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:43:02 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:33688
I20260806 03:43:02.633754 9346 raft_consensus.cc:2229] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:43:02.633874 9346 raft_consensus.cc:2258] T 3379bb4c17534e949a1583b36a7d6077 P 96b706fb5f2a411d8e3da5c06de965f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:02.634106 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8900
I20260806 03:43:02.636271 9348 tablet_replica.cc:330] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: stopping tablet replica
I20260806 03:43:02.636405 9348 raft_consensus.cc:2229] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:02.636561 9348 raft_consensus.cc:2258] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:02.636734 9347 tablet_replica.cc:330] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: stopping tablet replica
I20260806 03:43:02.636842 9347 raft_consensus.cc:2229] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:02.636946 9347 raft_consensus.cc:2258] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:02.637207 9348 ts_tablet_manager.cc:1905] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:43:02.637390 9347 ts_tablet_manager.cc:1905] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:43:02.639339 9348 ts_tablet_manager.cc:1918] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:43:02.639339 9347 ts_tablet_manager.cc:1918] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:43:02.639425 9348 log.cc:1196] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/3379bb4c17534e949a1583b36a7d6077
I20260806 03:43:02.639425 9347 log.cc:1196] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/3379bb4c17534e949a1583b36a7d6077
I20260806 03:43:02.639753 9348 ts_tablet_manager.cc:1939] T 3379bb4c17534e949a1583b36a7d6077 P 269339bafccf423f8c61271c681b26e8: Deleting consensus metadata
I20260806 03:43:02.639753 9347 ts_tablet_manager.cc:1939] T 3379bb4c17534e949a1583b36a7d6077 P fafbc67f030a40d3b9069e832e92aba2: Deleting consensus metadata
I20260806 03:43:02.640223 8822 catalog_manager.cc:4841] TS 269339bafccf423f8c61271c681b26e8 (127.30.24.2:39833): tablet 3379bb4c17534e949a1583b36a7d6077 (table test-table [id=d35815616a5a46c38c8e1be709f4f42c]) successfully deleted
I20260806 03:43:02.640312 8822 catalog_manager.cc:4841] TS fafbc67f030a40d3b9069e832e92aba2 (127.30.24.3:34873): tablet 3379bb4c17534e949a1583b36a7d6077 (table test-table [id=d35815616a5a46c38c8e1be709f4f42c]) successfully deleted
W20260806 03:43:02.641932 8820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:40723 (error 108)
I20260806 03:43:02.642216 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 9037
I20260806 03:43:02.643601 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:02.642358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:40723 (local address 127.0.0.1:49518)
0806 03:43:02.643529 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:02.643569 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:40723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1151}
W20260806 03:43:02.643775 8820 catalog_manager.cc:4584] TS 96b706fb5f2a411d8e3da5c06de965f2 (127.30.24.1:40723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3379bb4c17534e949a1583b36a7d6077: Network error: Client connection negotiation failed: client connection to 127.30.24.1:40723: connect: Connection refused (error 111)
I20260806 03:43:02.649361 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 9172
I20260806 03:43:02.654737 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 8803
2026-08-06T03:43:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1961 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[9352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[9352](info): set up 2 sockets
Aug 06 03:43:02 dist-test-slave-10pc krb5kdc[9352](info): commencing operation
krb5kdc: starting...
W20260806 03:43:03.707206 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:43:03 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:43:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/rangeradmin_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/rangeradmin_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/rangerlookup_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/rangerlookup_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
I20260806 03:43:03.885353 30816 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfG8t0l/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/postgres -l logfile start
2026-08-06 03:43:06.464 UTC [9383] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 03:43:06.464 UTC [9383] LOG: listening on IPv4 address "127.30.24.20", port 41819
2026-08-06 03:43:06.467 UTC [9383] LOG: listening on Unix socket "/tmp/.s.PGSQL.41819"
2026-08-06 03:43:06.489 UTC [9386] LOG: database system was shut down at 2026-08-06 03:43:05 UTC
2026-08-06 03:43:06.493 UTC [9383] LOG: database system is ready to accept connections
I20260806 03:43:07.458681 30816 mini_postgres.cc:96] Postgres bound to 41819
2026-08-06 03:43:07.467 UTC [9394] FATAL: database "slave" does not exist
127.30.24.20:41819 - accepting connections
I20260806 03:43:07.468297 30816 mini_ranger.cc:161] Starting Ranger...
I20260806 03:43:07.485551 30816 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 03:43:07.981755 30816 mini_ranger.cc:87] Created ranger Postgres database
I20260806 03:43:07.981848 30816 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-admin
2026-08-06 03:43:08,346 [I] DB FLAVOR :POSTGRES
2026-08-06 03:43:08,346 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:43:08,346 [I] Checking connection..
2026-08-06 03:43:08,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 03:43:08,626 [I] Checking connection passed.
2026-08-06 03:43:08,626 [I] --------- Verifying version history table ---------
2026-08-06 03:43:08,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:43:08,913 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:43:08,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:43:09,170 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:43:09,170 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 03:43:09,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 03:43:09.435 UTC [9458] WARNING: there is no transaction in progress
2026-08-06 03:43:09,442 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 03:43:09,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:43:09,711 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 03:43:09,711 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:43:09,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 03:43:09,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 03:43:10,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-10pc.c.gcp-upstream.internal','N') ;"
2026-08-06 03:43:10,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:43:10,756 [I] Table x_portal_user does not exist in database ranger
2026-08-06 03:43:10,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:43:11,005 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 03:43:11,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:43:11,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:43:11,504 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 03:43:11,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 03:43:12.481 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.499 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.516 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.532 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.549 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.568 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.625 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.631 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.645 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.655 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.666 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.680 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.690 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.696 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.703 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:12.710 UTC [9570] WARNING: there is no transaction in progress
2026-08-06 03:43:13,390 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 03:43:13,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 03:43:13,616 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 03:43:13,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:43:13,882 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 03:43:13,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:43:14,131 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 03:43:14,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:43:14,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:43:14,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-10pc.c.gcp-upstream.internal';"
2026-08-06 03:43:14,877 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 03:43:14,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 03:43:15,112 [I] DB_PATCHES have already been applied
I20260806 03:43:15.118287 30816 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfG8t0l/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 03:43:15.118371 30816 mini_ranger.cc:193] Using host: 127.30.24.20
I20260806 03:43:15.122763 30816 mini_ranger.cc:227] Ranger admin URL: http://127.30.24.20:40733
Aug 06, 2026 3:43:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:43:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:43:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 3:43:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 3:43:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 3:43:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 3:43:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 03:43:16.324618 30816 mini_ranger.cc:160] Time spent starting Ranger: real 8.856s user 0.005s sys 0.006s
Aug 06, 2026 3:43:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 3:43:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 3:43:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 3:43:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 3:43:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 03:43:38.454062 30816 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/rangerkms_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/rangerkms_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.24.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.24.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 03:43:38 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987818, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 03:43:38.568976 30816 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 03:43:38.579490 30816 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 03:43:39.453567 30816 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 03:43:39.453663 30816 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms
I20260806 03:43:39.453686 30816 mini_ranger_kms.cc:227] Using postgres at 127.30.24.20:41819
2026-08-06 03:43:39,982 [I] DB FLAVOR :POSTGRES
2026-08-06 03:43:39,982 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:43:39,982 [I] Checking connection
2026-08-06 03:43:39,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 03:43:40,230 [I] connection success
2026-08-06 03:43:40,231 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 03:43:40,231 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 03:43:40,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 03:43:40,483 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 03:43:40,483 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:43:40,483 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 03:43:40,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:41819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 03:43:40,741 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 03:43:41.224521 30816 mini_ranger_kms.cc:310] Created kms service
I20260806 03:43:41.430056 30816 mini_ranger_kms.cc:326] Created kudu user
I20260806 03:43:41.476701 30816 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 03:43:41 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987818, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:43:41.850584 30816 mini_ranger_kms.cc:384] Added ranger policy
I20260806 03:43:41.850705 30816 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 03:43:41.850728 30816 mini_ranger_kms.cc:240] Using host: 127.30.24.20
I20260806 03:43:41.854925 30816 mini_ranger_kms.cc:276] Ranger KMS PID: 9755
I20260806 03:43:41.855062 30816 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.24.20:59607
Aug 06, 2026 3:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 3:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 3:43:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 3:43:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 03:43:43 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785987823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.24.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:44.266314 30816 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.697s user 0.002s sys 0.263s
I20260806 03:43:44.266433 30816 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 03:43:44.266510 30816 mini_ranger_kms.cc:401] 127.30.24.20:59607/kms/v1/keys
Aug 06 03:43:44 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.24.20@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
Aug 06 03:43:44 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.24.20@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
Aug 06 03:43:47 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:43:47.226974 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:46403
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:46403
--ranger_config_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:59607/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:47.311292 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:47.311542 9807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:47.311580 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:47.312745 9807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:43:47.312813 9807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:47.312839 9807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:47.312872 9807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:43:47.312903 9807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:43:47.314519 9807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.24.20:59607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:46403
--ranger_config_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:47.314801 9807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:47.315037 9807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:43:47.320345 9807 server_base.cc:1034] running on GCE node
W20260806 03:43:47.320482 9812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:47.320667 9816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:47.320564 9813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:47.321045 9807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:47.321424 9807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:47.322587 9807 hybrid_clock.cc:648] HybridClock initialized: now 1785987827322571 us; error 36 us; skew 500 ppm
Aug 06 03:43:47 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987827, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:47.330219 9807 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:43:47.330789 9807 webserver.cc:460] Webserver started at http://127.30.24.62:44697/ using document root <none> and password file <none>
I20260806 03:43:47.331027 9807 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:47.331085 9807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:47.331251 9807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:43:47 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987827, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:43:47.388949 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "dc4e93805f844eaf9145ddeff212a6e7"
format_stamp: "Formatted at 2026-08-06 03:43:47 on dist-test-slave-10pc"
server_key: "10a7be7cac253674fc364350c9ea98c0"
server_key_iv: "ed9f324a6820a08a462bc63bd7acb6c3"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:47.389490 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "dc4e93805f844eaf9145ddeff212a6e7"
format_stamp: "Formatted at 2026-08-06 03:43:47 on dist-test-slave-10pc"
server_key: "10a7be7cac253674fc364350c9ea98c0"
server_key_iv: "ed9f324a6820a08a462bc63bd7acb6c3"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:47.391481 9807 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 03:43:47.408056 9823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:47.408325 9807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:43:47.408499 9807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "dc4e93805f844eaf9145ddeff212a6e7"
format_stamp: "Formatted at 2026-08-06 03:43:47 on dist-test-slave-10pc"
server_key: "10a7be7cac253674fc364350c9ea98c0"
server_key_iv: "ed9f324a6820a08a462bc63bd7acb6c3"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:47.408660 9807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:47.419688 9807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:47.421404 9807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:47.421532 9807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:47.425990 9807 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:46403
I20260806 03:43:47.426049 9877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:46403 every 8 connection(s)
I20260806 03:43:47.426424 9807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:43:47.427098 9878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:47.429478 9878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7: Bootstrap starting.
I20260806 03:43:47.430258 9878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:47.430521 9878 log.cc:824] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:47.431228 9878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7: No bootstrap required, opened a new log
I20260806 03:43:47.433431 9878 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e93805f844eaf9145ddeff212a6e7" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46403 } }
I20260806 03:43:47.433593 9878 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:47.433617 9878 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4e93805f844eaf9145ddeff212a6e7, State: Initialized, Role: FOLLOWER
I20260806 03:43:47.433725 9878 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e93805f844eaf9145ddeff212a6e7" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46403 } }
I20260806 03:43:47.433780 9878 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:43:47.433800 9878 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:43:47.433880 9878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:47.434077 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 9807
I20260806 03:43:47.434206 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:43:47.434397 30816 external_mini_cluster.cc:1428] Setting key 3a8d9456860f1c5ed61c697ae3c0b2ea
I20260806 03:43:47.434605 9878 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e93805f844eaf9145ddeff212a6e7" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46403 } }
I20260806 03:43:47.434731 9878 leader_election.cc:302] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc4e93805f844eaf9145ddeff212a6e7; no voters:
I20260806 03:43:47.434930 9878 leader_election.cc:288] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:43:47.435103 9884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:43:47.435212 9878 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:43:47.435256 9884 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [term 1 LEADER]: Becoming Leader. State: Replica: dc4e93805f844eaf9145ddeff212a6e7, State: Running, Role: LEADER
I20260806 03:43:47.435374 9884 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e93805f844eaf9145ddeff212a6e7" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46403 } }
I20260806 03:43:47.435679 9884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4e93805f844eaf9145ddeff212a6e7. Latest consensus state: current_term: 1 leader_uuid: "dc4e93805f844eaf9145ddeff212a6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e93805f844eaf9145ddeff212a6e7" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46403 } } }
I20260806 03:43:47.435689 9885 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc4e93805f844eaf9145ddeff212a6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e93805f844eaf9145ddeff212a6e7" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46403 } } }
I20260806 03:43:47.435865 9885 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [sys.catalog]: This master's current role is: LEADER
I20260806 03:43:47.435812 9884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7 [sys.catalog]: This master's current role is: LEADER
I20260806 03:43:47.436214 9893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:43:47.436848 9893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:43:47.437386 9878 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 03:43:47.442762 9893 catalog_manager.cc:1347] Generated new cluster ID: 3eba753e6a5141d7b62019f3a031cd32
I20260806 03:43:47.442939 9893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:43:47 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:47.449767 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:47.434898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.30.24.62:46403)
0806 03:43:47.435938 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:47.435948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:47.435988 (+ 40us) server_negotiation.cc:407] Connection header received
0806 03:43:47.436114 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:47.436122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:47.436193 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:47.436343 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:47.437627 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:47.441302 (+ 3675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:47.442170 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:47.443226 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:47.443397 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:47.446243 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:47.446247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:47.446247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:47.446259 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:47.448069 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:47.448535 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:47.448538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:47.448538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:47.448631 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:47.448844 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:47.448845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:47.448845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:47.449105 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:47.449295 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:47.449416 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 03:43:47.449551 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":79,"threads_started":1}
I20260806 03:43:47.453437 9893 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:43:47.454169 9893 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:43:47.464177 9893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7: Generated new TSK 0
I20260806 03:43:47.464370 9893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:43:48.183890 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:59607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46403
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:48.288924 9925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:48.289181 9925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:48.289209 9925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:48.290540 9925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:48.290593 9925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:48.290622 9925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:43:48.292340 9925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.24.20:59607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:48.292673 9925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:48.292955 9925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:48.298681 9934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:48.298803 9932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:48.299072 9931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:48.299253 9925 server_base.cc:1034] running on GCE node
I20260806 03:43:48.299465 9925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:48.299754 9925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:48.300927 9925 hybrid_clock.cc:648] HybridClock initialized: now 1785987828300903 us; error 36 us; skew 500 ppm
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:48.304437 9925 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:43:48.305032 9925 webserver.cc:460] Webserver started at http://127.30.24.1:36089/ using document root <none> and password file <none>
I20260806 03:43:48.305365 9925 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:48.305444 9925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:48.305598 9925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:43:48.322391 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "b0abc12020c0488b9f3fe39c0f9f1469"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "3aa5469e1d553f0aa8f20a7d65af6a8b"
server_key_iv: "c9d5fe2c1ccc5130458a00cfd2e4e7c5"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.322872 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "b0abc12020c0488b9f3fe39c0f9f1469"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "3aa5469e1d553f0aa8f20a7d65af6a8b"
server_key_iv: "c9d5fe2c1ccc5130458a00cfd2e4e7c5"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.326335 9925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 03:43:48.340562 9941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.340798 9925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:43:48.340911 9925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "b0abc12020c0488b9f3fe39c0f9f1469"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "3aa5469e1d553f0aa8f20a7d65af6a8b"
server_key_iv: "c9d5fe2c1ccc5130458a00cfd2e4e7c5"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.341055 9925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:48.382938 9925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:48.385980 9925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:48.386261 9925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:48.386628 9925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:48.387140 9925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:48.387251 9925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.387368 9925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:48.387458 9925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.395608 9925 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:37617
I20260806 03:43:48.395689 10058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:37617 every 8 connection(s)
I20260806 03:43:48.396154 9925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:48.403152 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 9925
I20260806 03:43:48.403290 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:43:48.403510 30816 external_mini_cluster.cc:1428] Setting key 108f6cb4377f152082d820574f8540a1
I20260806 03:43:48.411942 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:48.397017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:38241 (local address 127.30.24.62:46403)
0806 03:43:48.397189 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:48.397191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:48.397318 (+ 127us) server_negotiation.cc:407] Connection header received
0806 03:43:48.397856 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:48.397858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:48.397873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:48.397953 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:48.398628 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.399146 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.400151 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.400366 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.400437 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:48.408045 (+ 7608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:48.408048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:48.408048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:48.408056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:48.410155 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:48.410701 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:48.410703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:48.410703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:48.410769 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:48.411078 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:48.411085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:48.411085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:48.411259 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:48.411305 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:48.411742 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 03:43:48.411834 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260806 03:43:48.412281 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:48.396993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46403 (local address 127.30.24.1:38241)
0806 03:43:48.397184 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:48.397233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:48.397751 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:48.398029 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:48.398032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:48.398271 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:48.398531 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:48.398535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.399257 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.399258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:48.400049 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:48.400052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.400445 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.400446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:48.400610 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:48.401229 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:43:48.401239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:48.407913 (+ 6674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:48.410263 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:48.410273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:48.410276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:48.410607 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:48.410840 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:48.410841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:48.410841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:48.410985 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:48.411358 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:48.411365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:48.411560 (+ 195us) client_negotiation.cc:769] Sending connection context
0806 03:43:48.411703 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 03:43:48.411875 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260806 03:43:48.412835 10059 heartbeater.cc:347] Connected to a master server at 127.30.24.62:46403
I20260806 03:43:48.412940 10059 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:48.413146 10059 heartbeater.cc:511] Master 127.30.24.62:46403 requested a full tablet report, sending...
I20260806 03:43:48.413853 9842 ts_manager.cc:194] Registered new tserver with Master: b0abc12020c0488b9f3fe39c0f9f1469 (127.30.24.1:37617)
I20260806 03:43:48.414783 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:38241
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:43:48.437721 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:59607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46403
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 03:43:48.561085 10066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:48.561404 10066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:48.561446 10066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:48.563262 10066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:48.563329 10066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:48.563371 10066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:43:48.565750 10066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.24.20:59607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:48.566028 10066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:48.566308 10066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:48.573591 10074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:48.573587 10076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:48.573805 10073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:48.574923 10066 server_base.cc:1034] running on GCE node
I20260806 03:43:48.575181 10066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:48.575451 10066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:48.576619 10066 hybrid_clock.cc:648] HybridClock initialized: now 1785987828576595 us; error 34 us; skew 500 ppm
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:48.580044 10066 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:43:48.580637 10066 webserver.cc:460] Webserver started at http://127.30.24.2:46231/ using document root <none> and password file <none>
I20260806 03:43:48.580919 10066 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:48.580981 10066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:48.581113 10066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:43:48.598905 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "43c613cc2ca946228f771b5cce3a4116"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "ce2675a42cf41ad3c51d6ada851e30e8"
server_key_iv: "7e912d85f7199b1edb965e4fe9c4fdce"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.599362 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "43c613cc2ca946228f771b5cce3a4116"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "ce2675a42cf41ad3c51d6ada851e30e8"
server_key_iv: "7e912d85f7199b1edb965e4fe9c4fdce"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.604673 10066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260806 03:43:48.620748 10084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.620985 10066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 03:43:48.621124 10066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "43c613cc2ca946228f771b5cce3a4116"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "ce2675a42cf41ad3c51d6ada851e30e8"
server_key_iv: "7e912d85f7199b1edb965e4fe9c4fdce"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.621239 10066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:48.651717 10066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:48.654527 10066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:48.654713 10066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:48.654944 10066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:48.655340 10066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:48.655396 10066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.655435 10066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:48.655464 10066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.662396 10066 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:36205
I20260806 03:43:48.662961 10066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:43:48.664808 10199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:36205 every 8 connection(s)
I20260806 03:43:48.666136 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 10066
I20260806 03:43:48.666254 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:43:48.666469 30816 external_mini_cluster.cc:1428] Setting key e40c5f8e06de30f9ef3740f0af341ac2
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:48.687335 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:48.667642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:46321 (local address 127.30.24.62:46403)
0806 03:43:48.667690 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:48.667693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:48.673795 (+ 6102us) server_negotiation.cc:407] Connection header received
0806 03:43:48.674716 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:48.674718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:48.674732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:48.674765 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:48.675730 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.676582 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.678132 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.678405 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.678475 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:48.683736 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:48.683738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:48.683738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:48.683747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:48.685582 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:48.686129 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:48.686131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:48.686131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:48.686195 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:48.686498 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:48.686499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:48.686499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:48.686648 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:48.686705 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:48.687140 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 03:43:48.687234 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:43:48.688176 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:48.667350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46403 (local address 127.30.24.2:46321)
0806 03:43:48.673598 (+ 6248us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:48.673670 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:48.674564 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:48.674985 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:48.674991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:48.675288 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:48.675604 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:48.675610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.676709 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.676710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:48.678021 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:48.678024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.678550 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.678551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:48.678971 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:48.679607 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:43:48.679619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:48.683594 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:48.685705 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:48.685713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:48.685716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:48.686030 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:48.686271 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:48.686272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:48.686272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:48.686415 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:48.686758 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:48.686764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:48.686974 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 03:43:48.687727 (+ 753us) client_negotiation.cc:240] Negotiation successful
0806 03:43:48.687927 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6179,"thread_start_us":95,"threads_started":1}
I20260806 03:43:48.688944 10200 heartbeater.cc:347] Connected to a master server at 127.30.24.62:46403
I20260806 03:43:48.689073 10200 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:48.689277 10200 heartbeater.cc:511] Master 127.30.24.62:46403 requested a full tablet report, sending...
I20260806 03:43:48.689680 9842 ts_manager.cc:194] Registered new tserver with Master: 43c613cc2ca946228f771b5cce3a4116 (127.30.24.2:36205)
I20260806 03:43:48.690181 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:46321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:43:48.717857 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:59607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46403
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:43:48.849411 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:43:48.849646 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:43:48.849695 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:43:48.850904 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:43:48.850971 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:43:48.851019 10205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:43:48.853250 10205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.24.20:59607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:43:48.853579 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:43:48.853919 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:43:48.870437 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:43:48.870920 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:48.872411 10205 server_base.cc:1034] running on GCE node
W20260806 03:43:48.874965 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:43:48.875243 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:43:48.875563 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:43:48.876727 10205 hybrid_clock.cc:648] HybridClock initialized: now 1785987828876703 us; error 38 us; skew 500 ppm
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:43:48.880283 10205 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:43:48.880966 10205 webserver.cc:460] Webserver started at http://127.30.24.3:46033/ using document root <none> and password file <none>
I20260806 03:43:48.881304 10205 fs_manager.cc:359] Metadata directory not provided
I20260806 03:43:48.881419 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:43:48.881606 10205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:43:48.898240 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "9611c604acec4923af77fbaffa3502f5"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "085197f395c6a2648d2372e8e519540d"
server_key_iv: "525af83218196e9c2bd8707d812a0525"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.898963 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "9611c604acec4923af77fbaffa3502f5"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "085197f395c6a2648d2372e8e519540d"
server_key_iv: "525af83218196e9c2bd8707d812a0525"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.913435 10205 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.000s sys 0.002s
I20260806 03:43:48.929257 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.929425 10205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:43:48.929500 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "9611c604acec4923af77fbaffa3502f5"
format_stamp: "Formatted at 2026-08-06 03:43:48 on dist-test-slave-10pc"
server_key: "085197f395c6a2648d2372e8e519540d"
server_key_iv: "525af83218196e9c2bd8707d812a0525"
server_key_version: "kuduclusterkey@0"
I20260806 03:43:48.929589 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:43:48.965664 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:43:48.968328 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:43:48.968523 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:43:48.968884 10205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:43:48.969605 10205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:43:48.969673 10205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.969710 10205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:43:48.969732 10205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:43:48.977450 10205 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:44017
I20260806 03:43:48.977957 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:43:48.978363 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 10205
I20260806 03:43:48.978482 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:43:48.978673 30816 external_mini_cluster.cc:1428] Setting key 227bbdd9bfec884ea70958c2cf337e27
I20260806 03:43:48.982470 10335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:44017 every 8 connection(s)
Aug 06 03:43:48 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:43:48.993992 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:48.982927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:45589 (local address 127.30.24.62:46403)
0806 03:43:48.982975 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:48.982977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:48.983197 (+ 220us) server_negotiation.cc:407] Connection header received
0806 03:43:48.983897 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:48.983899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:48.983912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:48.983942 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:48.984845 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.985582 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.987026 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.987284 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.987324 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:48.990109 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:48.990110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:48.990110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:48.990118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:48.991994 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:48.992607 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:48.992608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:48.992608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:48.992677 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:48.993009 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:48.993011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:48.993011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:48.993163 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:48.993226 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:48.993776 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 03:43:48.993874 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:43:48.994982 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:48.982742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46403 (local address 127.30.24.3:45589)
0806 03:43:48.983063 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:48.983114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:48.983762 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:48.984153 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:48.984158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:48.984404 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:43:48.984733 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:48.984739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.985701 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.985702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:48.986928 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:48.986931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:48.987414 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:48.987415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:48.987563 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:48.988196 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:43:48.988207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:43:48.989996 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:43:48.992104 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:48.992112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:48.992115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:48.992448 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:48.992759 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:43:48.992760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:43:48.992761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:43:48.992921 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:43:48.993267 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:43:48.993274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:43:48.993587 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 03:43:48.994500 (+ 913us) client_negotiation.cc:240] Negotiation successful
0806 03:43:48.994715 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260806 03:43:48.995627 10336 heartbeater.cc:347] Connected to a master server at 127.30.24.62:46403
I20260806 03:43:48.995720 10336 heartbeater.cc:464] Registering TS with master...
I20260806 03:43:48.995910 10336 heartbeater.cc:511] Master 127.30.24.62:46403 requested a full tablet report, sending...
I20260806 03:43:48.996312 9842 ts_manager.cc:194] Registered new tserver with Master: 9611c604acec4923af77fbaffa3502f5 (127.30.24.3:44017)
I20260806 03:43:48.996737 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:45589
I20260806 03:43:49.001929 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:43:49.014293 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.005984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.30.24.62:46403)
0806 03:43:49.006033 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.006048 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.006357 (+ 309us) server_negotiation.cc:407] Connection header received
0806 03:43:49.006453 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.006453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.006465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.006494 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:43:49.007295 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.007991 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.009110 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.009382 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.009421 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.011084 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:43:49.011086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:43:49.011086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:43:49.011093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:43:49.013010 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:49.013436 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:49.013437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:49.013437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:49.013493 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:43:49.013700 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:43:49.013701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:43:49.013701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:43:49.013823 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:43:49.013894 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.014114 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.014195 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:43:49.016530 9842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:43:49.018700 9842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:43:49.038096 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.024136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36205 (local address 127.0.0.1:54558)
0806 03:43:49.025936 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:49.025981 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:49.026046 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:49.026587 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:49.026591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:49.026593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:49.026669 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.026671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.034880 (+ 8209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.034881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.035926 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.035929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.037781 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.037782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.037866 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.037882 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 03:43:49.037951 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 03:43:49.038002 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":77,"threads_started":1}
I20260806 03:43:49.041165 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.024369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37617 (local address 127.0.0.1:60760)
0806 03:43:49.026148 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:49.026156 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:49.026190 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:49.030115 (+ 3925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:49.030116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:49.030118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:49.030164 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.030166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.032712 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.032713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.034278 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.034281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.040934 (+ 6653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.040935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.041000 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.041005 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:43:49.041047 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 03:43:49.041088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1766,"thread_start_us":64,"threads_started":1}
I20260806 03:43:49.041679 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.025013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.30.24.2:36205)
0806 03:43:49.026330 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.026336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.026346 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:43:49.026385 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.026388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.026402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.026472 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:49.026763 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.034797 (+ 8034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.036044 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.037596 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.037661 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.037745 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.041550 (+ 3805us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.041601 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":110,"threads_started":1}
I20260806 03:43:49.042243 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.025229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.30.24.1:37617)
0806 03:43:49.029903 (+ 4674us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.029908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.029918 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:43:49.029952 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.029954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.029969 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.030037 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:49.031503 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.032615 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.039753 (+ 7138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.040506 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.040583 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.040642 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.042122 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.042175 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4640,"thread_start_us":57,"threads_started":1}
I20260806 03:43:49.042541 10132 tablet_service.cc:1467] Processing CreateTablet for tablet af2de8c2d2434e98909057125b535959 (DEFAULT_TABLE table=test-table [id=3cf021b6d4bd483680b9b06917b6c6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:43:49.042945 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2de8c2d2434e98909057125b535959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:49.045154 9973 tablet_service.cc:1467] Processing CreateTablet for tablet af2de8c2d2434e98909057125b535959 (DEFAULT_TABLE table=test-table [id=3cf021b6d4bd483680b9b06917b6c6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:43:49.045472 9973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2de8c2d2434e98909057125b535959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:49.050136 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.024545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:44017 (local address 127.0.0.1:44862)
0806 03:43:49.029730 (+ 5185us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:49.029741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:49.029795 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:49.037144 (+ 7349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:49.037145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:49.037148 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:49.037193 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.037195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.042482 (+ 5287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.042484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.043796 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.043799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.049892 (+ 6093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.049893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.049961 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.049966 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:43:49.050012 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 03:43:49.050061 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5161,"thread_start_us":63,"threads_started":1}
I20260806 03:43:49.050582 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.025374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.30.24.3:44017)
0806 03:43:49.036904 (+ 11530us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.036910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.036922 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:43:49.036963 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.036965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.036982 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.037058 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:49.038552 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.039606 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.043914 (+ 4308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.049702 (+ 5788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.049774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.049854 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.050446 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.050494 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11483,"thread_start_us":91,"threads_started":1}
I20260806 03:43:49.051295 10263 tablet_service.cc:1467] Processing CreateTablet for tablet af2de8c2d2434e98909057125b535959 (DEFAULT_TABLE table=test-table [id=3cf021b6d4bd483680b9b06917b6c6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:43:49.051616 10263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2de8c2d2434e98909057125b535959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:49.055694 10358 tablet_bootstrap.cc:492] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: Bootstrap starting.
I20260806 03:43:49.056586 10358 tablet_bootstrap.cc:654] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:49.056810 10359 tablet_bootstrap.cc:492] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Bootstrap starting.
I20260806 03:43:49.057860 10359 tablet_bootstrap.cc:654] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:49.058259 10358 log.cc:824] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:49.058847 10359 log.cc:824] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:49.059250 10357 tablet_bootstrap.cc:492] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Bootstrap starting.
I20260806 03:43:49.060379 10357 tablet_bootstrap.cc:654] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Neither blocks nor log segments found. Creating new log.
I20260806 03:43:49.060760 10357 log.cc:824] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Log is configured to *not* fsync() on all Append() calls
I20260806 03:43:49.061594 10358 tablet_bootstrap.cc:492] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: No bootstrap required, opened a new log
I20260806 03:43:49.061713 10358 ts_tablet_manager.cc:1397] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260806 03:43:49.064798 10358 raft_consensus.cc:348] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.065114 10358 raft_consensus.cc:374] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:49.065196 10358 raft_consensus.cc:729] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0abc12020c0488b9f3fe39c0f9f1469, State: Initialized, Role: FOLLOWER
I20260806 03:43:49.065387 10358 consensus_queue.cc:260] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.065768 10358 ts_tablet_manager.cc:1428] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260806 03:43:49.065920 10059 heartbeater.cc:503] Master 127.30.24.62:46403 was elected leader, sending a full tablet report...
I20260806 03:43:49.067449 10359 tablet_bootstrap.cc:492] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: No bootstrap required, opened a new log
I20260806 03:43:49.067588 10359 ts_tablet_manager.cc:1397] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260806 03:43:49.069741 10359 raft_consensus.cc:348] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.069994 10359 raft_consensus.cc:374] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:49.070073 10359 raft_consensus.cc:729] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9611c604acec4923af77fbaffa3502f5, State: Initialized, Role: FOLLOWER
I20260806 03:43:49.070254 10359 consensus_queue.cc:260] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.070573 10359 ts_tablet_manager.cc:1428] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:43:49.070725 10336 heartbeater.cc:503] Master 127.30.24.62:46403 was elected leader, sending a full tablet report...
I20260806 03:43:49.072012 10357 tablet_bootstrap.cc:492] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: No bootstrap required, opened a new log
I20260806 03:43:49.072140 10357 ts_tablet_manager.cc:1397] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260806 03:43:49.074545 10357 raft_consensus.cc:348] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.074774 10357 raft_consensus.cc:374] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:43:49.074853 10357 raft_consensus.cc:729] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c613cc2ca946228f771b5cce3a4116, State: Initialized, Role: FOLLOWER
I20260806 03:43:49.076889 10357 consensus_queue.cc:260] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.077201 10357 ts_tablet_manager.cc:1428] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 03:43:49.077312 10200 heartbeater.cc:503] Master 127.30.24.62:46403 was elected leader, sending a full tablet report...
Aug 06 03:43:49 dist-test-slave-10pc krb5kdc[9352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987828, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
W20260806 03:43:49.147152 10060 tablet.cc:2369] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:43:49.168192 10201 tablet.cc:2369] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:43:49.215936 10363 raft_consensus.cc:482] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.216115 10363 raft_consensus.cc:504] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.216562 10363 leader_election.cc:288] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c613cc2ca946228f771b5cce3a4116 (127.30.24.2:36205), 9611c604acec4923af77fbaffa3502f5 (127.30.24.3:44017)
I20260806 03:43:49.220563 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.217140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:52985 (local address 127.30.24.3:44017)
0806 03:43:49.217187 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.217189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.217322 (+ 133us) server_negotiation.cc:407] Connection header received
0806 03:43:49.217357 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.217359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.217372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.217431 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:49.217638 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.218127 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.219692 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.220200 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.220282 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.220293 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.220428 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.220493 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:43:49.220598 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.217068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:44017 (local address 127.30.24.1:52985)
0806 03:43:49.217232 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:49.217243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:49.217306 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:49.217431 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:49.217432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:49.217433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:49.217506 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.217507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.218241 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.218242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.219594 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.219597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.220286 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.220287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.220344 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.220348 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:43:49.220416 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 03:43:49.220477 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":54,"thread_start_us":77,"threads_started":1}
I20260806 03:43:49.221056 10271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2de8c2d2434e98909057125b535959" candidate_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9611c604acec4923af77fbaffa3502f5" is_pre_election: true
I20260806 03:43:49.221230 10271 raft_consensus.cc:2454] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0abc12020c0488b9f3fe39c0f9f1469 in term 0.
I20260806 03:43:49.221493 9943 leader_election.cc:302] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9611c604acec4923af77fbaffa3502f5, b0abc12020c0488b9f3fe39c0f9f1469; no voters:
I20260806 03:43:49.221593 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.216862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:56611 (local address 127.30.24.2:36205)
0806 03:43:49.216928 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.216931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.217118 (+ 187us) server_negotiation.cc:407] Connection header received
0806 03:43:49.217252 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.217254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.217269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.217329 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:43:49.217548 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.218204 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.220330 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.220849 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.220922 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.220931 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.221470 (+ 539us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.221523 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:43:49.221606 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.216780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36205 (local address 127.30.24.1:56611)
0806 03:43:49.217023 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:49.217037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:43:49.217150 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:43:49.217378 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:43:49.217379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:43:49.217381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:43:49.217473 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.217475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.218273 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.218274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.219594 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:43:49.219597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.221309 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.221311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:43:49.221386 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.221391 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:43:49.221465 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 03:43:49.221505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":80,"thread_start_us":152,"threads_started":1}
I20260806 03:43:49.221658 10363 raft_consensus.cc:2781] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:43:49.221719 10363 raft_consensus.cc:482] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.221741 10363 raft_consensus.cc:3037] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:49.222442 10363 raft_consensus.cc:504] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.222532 10363 leader_election.cc:288] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [CANDIDATE]: Term 1 election: Requested vote from peers 43c613cc2ca946228f771b5cce3a4116 (127.30.24.2:36205), 9611c604acec4923af77fbaffa3502f5 (127.30.24.3:44017)
I20260806 03:43:49.222996 10271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2de8c2d2434e98909057125b535959" candidate_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9611c604acec4923af77fbaffa3502f5"
I20260806 03:43:49.223104 10271 raft_consensus.cc:3037] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:49.223042 10151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2de8c2d2434e98909057125b535959" candidate_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c613cc2ca946228f771b5cce3a4116"
I20260806 03:43:49.223048 10152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2de8c2d2434e98909057125b535959" candidate_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c613cc2ca946228f771b5cce3a4116" is_pre_election: true
I20260806 03:43:49.223197 10151 raft_consensus.cc:3037] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:43:49.224056 10271 raft_consensus.cc:2454] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0abc12020c0488b9f3fe39c0f9f1469 in term 1.
I20260806 03:43:49.224062 10151 raft_consensus.cc:2454] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0abc12020c0488b9f3fe39c0f9f1469 in term 1.
I20260806 03:43:49.224222 9943 leader_election.cc:302] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c613cc2ca946228f771b5cce3a4116, b0abc12020c0488b9f3fe39c0f9f1469; no voters:
I20260806 03:43:49.224345 10363 raft_consensus.cc:2781] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:43:49.224525 10363 raft_consensus.cc:686] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 1 LEADER]: Becoming Leader. State: Replica: b0abc12020c0488b9f3fe39c0f9f1469, State: Running, Role: LEADER
I20260806 03:43:49.224647 10363 consensus_queue.cc:237] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } }
I20260806 03:43:49.225572 9842 catalog_manager.cc:5495] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0abc12020c0488b9f3fe39c0f9f1469 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0abc12020c0488b9f3fe39c0f9f1469" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 } health_report { overall_health: UNKNOWN } } }
W20260806 03:43:49.232327 10337 tablet.cc:2369] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:43:49.258572 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:43:49.255207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.30.24.1:37617)
0806 03:43:49.255322 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 03:43:49.255324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:43:49.255331 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:43:49.255372 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:43:49.255373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:43:49.255397 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:43:49.255467 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:43:49.255648 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.256376 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.257513 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:43:49.257831 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:43:49.257905 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:43:49.258137 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:43:49.258289 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:43:49.258389 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 03:43:49.258442 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 03:43:49.258466 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 03:43:49.260846 10151 raft_consensus.cc:1261] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 1 FOLLOWER]: Refusing update from remote peer b0abc12020c0488b9f3fe39c0f9f1469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:49.260846 10271 raft_consensus.cc:1261] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 1 FOLLOWER]: Refusing update from remote peer b0abc12020c0488b9f3fe39c0f9f1469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:49.261183 10363 consensus_queue.cc:1035] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9611c604acec4923af77fbaffa3502f5" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:43:49.261353 10368 consensus_queue.cc:1035] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c613cc2ca946228f771b5cce3a4116" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:43:49.263865 10371 mvcc.cc:205] Tried to move back new op lower bound from 7315406148649328640 to 7315406148504887296. Current Snapshot: MvccSnapshot[applied={T|T < 7315406148649328640}]
I20260806 03:43:49.264889 10372 mvcc.cc:205] Tried to move back new op lower bound from 7315406148649328640 to 7315406148504887296. Current Snapshot: MvccSnapshot[applied={T|T < 7315406148649328640}]
I20260806 03:43:49.269955 10373 mvcc.cc:205] Tried to move back new op lower bound from 7315406148649328640 to 7315406148504887296. Current Snapshot: MvccSnapshot[applied={T|T < 7315406148649328640}]
I20260806 03:43:49.271117 9841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:43:49.271214 9841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:43:49.271998 9841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc4e93805f844eaf9145ddeff212a6e7: Sending DeleteTablet for 3 replicas of tablet af2de8c2d2434e98909057125b535959
I20260806 03:43:49.272516 10132 tablet_service.cc:1514] Processing DeleteTablet for tablet af2de8c2d2434e98909057125b535959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:43:49 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:54558
I20260806 03:43:49.272789 9973 tablet_service.cc:1514] Processing DeleteTablet for tablet af2de8c2d2434e98909057125b535959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:43:49 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:60760
I20260806 03:43:49.273079 10263 tablet_service.cc:1514] Processing DeleteTablet for tablet af2de8c2d2434e98909057125b535959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:43:49 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:44862
I20260806 03:43:49.273677 10380 tablet_replica.cc:330] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: stopping tablet replica
I20260806 03:43:49.273810 10381 tablet_replica.cc:330] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469: stopping tablet replica
I20260806 03:43:49.273878 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 9925
I20260806 03:43:49.273931 10382 tablet_replica.cc:330] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: stopping tablet replica
I20260806 03:43:49.274019 10382 raft_consensus.cc:2229] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:49.274127 10382 raft_consensus.cc:2258] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:49.274247 10381 raft_consensus.cc:2229] T af2de8c2d2434e98909057125b535959 P b0abc12020c0488b9f3fe39c0f9f1469 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:43:49.274526 10380 raft_consensus.cc:2229] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:49.274638 10380 raft_consensus.cc:2258] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:49.275306 10382 ts_tablet_manager.cc:1905] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:43:49.277129 10382 ts_tablet_manager.cc:1918] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:43:49.277196 10382 log.cc:1196] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/af2de8c2d2434e98909057125b535959
I20260806 03:43:49.277755 10382 ts_tablet_manager.cc:1939] T af2de8c2d2434e98909057125b535959 P 9611c604acec4923af77fbaffa3502f5: Deleting consensus metadata
I20260806 03:43:49.278190 9825 catalog_manager.cc:4841] TS 9611c604acec4923af77fbaffa3502f5 (127.30.24.3:44017): tablet af2de8c2d2434e98909057125b535959 (table test-table [id=3cf021b6d4bd483680b9b06917b6c6a3]) successfully deleted
I20260806 03:43:49.278757 10380 ts_tablet_manager.cc:1905] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:43:49.280501 10380 ts_tablet_manager.cc:1918] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:43:49.280582 10380 log.cc:1196] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/af2de8c2d2434e98909057125b535959
I20260806 03:43:49.280843 10380 ts_tablet_manager.cc:1939] T af2de8c2d2434e98909057125b535959 P 43c613cc2ca946228f771b5cce3a4116: Deleting consensus metadata
I20260806 03:43:49.281432 9825 catalog_manager.cc:4841] TS 43c613cc2ca946228f771b5cce3a4116 (127.30.24.2:36205): tablet af2de8c2d2434e98909057125b535959 (table test-table [id=3cf021b6d4bd483680b9b06917b6c6a3]) successfully deleted
W20260806 03:43:49.285481 9825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:37617 (error 108)
W20260806 03:43:49.286101 10351 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:43:49.285949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:37617 (local address 127.0.0.1:60778)
0806 03:43:49.286011 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 03:43:49.286053 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:37617: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 03:43:49.286223 9825 catalog_manager.cc:4584] TS b0abc12020c0488b9f3fe39c0f9f1469 (127.30.24.1:37617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af2de8c2d2434e98909057125b535959: Network error: Client connection negotiation failed: client connection to 127.30.24.1:37617: connect: Connection reset by peer (error 104)
I20260806 03:43:49.286583 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 10066
I20260806 03:43:49.294042 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 10205
I20260806 03:43:49.302001 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 9807
I20260806 03:43:49.308758 30816 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 03:43:49.653204 30816 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 03:43:49.653314 30816 mini_ranger.cc:66] Stopping Ranger...
I20260806 03:43:50.008010 30816 mini_ranger.cc:68] Stopped Ranger
2026-08-06 03:43:50.008 UTC [9383] LOG: received smart shutdown request
2026-08-06 03:43:50.012 UTC [9383] LOG: background worker "logical replication launcher" (PID 9392) exited with exit code 1
2026-08-06 03:43:50.012 UTC [9387] LOG: shutting down
2026-08-06 03:43:50.172 UTC [9383] LOG: database system is shut down
2026-08-06T03:43:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47638 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:43:50 dist-test-slave-10pc krb5kdc[10392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:43:50 dist-test-slave-10pc krb5kdc[10392](info): set up 2 sockets
Aug 06 03:43:50 dist-test-slave-10pc krb5kdc[10392](info): commencing operation
krb5kdc: starting...
W20260806 03:43:51.353820 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:43:51 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:43:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/rangeradmin_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/rangeradmin_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/rangerlookup_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/rangerlookup_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
I20260806 03:43:51.527536 30816 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfG8t0l/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/postgres -l logfile start
2026-08-06 03:43:54.051 UTC [10424] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 03:43:54.051 UTC [10424] LOG: listening on IPv4 address "127.30.24.20", port 57701
2026-08-06 03:43:54.055 UTC [10424] LOG: listening on Unix socket "/tmp/.s.PGSQL.57701"
2026-08-06 03:43:54.075 UTC [10427] LOG: database system was shut down at 2026-08-06 03:43:52 UTC
2026-08-06 03:43:54.079 UTC [10424] LOG: database system is ready to accept connections
I20260806 03:43:55.081537 30816 mini_postgres.cc:96] Postgres bound to 57701
2026-08-06 03:43:55.088 UTC [10435] FATAL: database "slave" does not exist
127.30.24.20:57701 - accepting connections
I20260806 03:43:55.089274 30816 mini_ranger.cc:161] Starting Ranger...
I20260806 03:43:55.104605 30816 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 03:43:55.569345 30816 mini_ranger.cc:87] Created ranger Postgres database
I20260806 03:43:55.569416 30816 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-admin
2026-08-06 03:43:55,836 [I] DB FLAVOR :POSTGRES
2026-08-06 03:43:55,836 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:43:55,836 [I] Checking connection..
2026-08-06 03:43:55,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 03:43:56,086 [I] Checking connection passed.
2026-08-06 03:43:56,086 [I] --------- Verifying version history table ---------
2026-08-06 03:43:56,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:43:56,348 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:43:56,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:43:56,600 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:43:56,601 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 03:43:56,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 03:43:56.865 UTC [10500] WARNING: there is no transaction in progress
2026-08-06 03:43:56,873 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 03:43:56,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:43:57,125 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 03:43:57,125 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:43:57,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 03:43:57,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 03:43:57,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-10pc.c.gcp-upstream.internal','N') ;"
2026-08-06 03:43:57,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:43:58,176 [I] Table x_portal_user does not exist in database ranger
2026-08-06 03:43:58,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:43:58,429 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 03:43:58,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:43:58,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:43:58,914 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 03:43:58,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 03:43:59.880 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:43:59.897 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:43:59.913 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:43:59.928 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:43:59.946 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:43:59.963 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.018 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.025 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.039 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.050 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.060 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.075 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.084 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.090 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.097 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00.104 UTC [10612] WARNING: there is no transaction in progress
2026-08-06 03:44:00,754 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 03:44:00,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 03:44:00,981 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 03:44:00,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:44:01,235 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 03:44:01,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:44:01,467 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 03:44:01,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:44:01,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:44:01,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-10pc.c.gcp-upstream.internal';"
2026-08-06 03:44:02,198 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 03:44:02,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 03:44:02,440 [I] DB_PATCHES have already been applied
I20260806 03:44:02.446219 30816 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfG8t0l/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 03:44:02.446298 30816 mini_ranger.cc:193] Using host: 127.30.24.20
I20260806 03:44:02.450600 30816 mini_ranger.cc:227] Ranger admin URL: http://127.30.24.20:46531
Aug 06, 2026 3:44:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:44:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:44:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 3:44:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 3:44:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 3:44:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 3:44:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 03:44:03.589542 30816 mini_ranger.cc:160] Time spent starting Ranger: real 8.500s user 0.006s sys 0.005s
Aug 06, 2026 3:44:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 3:44:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 3:44:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 3:44:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 3:44:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 03:44:24.942584 30816 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.24.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.24.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/rangerkms_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.24.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/rangerkms_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.24.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.24.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.24.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/HTTP_127.30.24.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 03:44:25 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987865, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 03:44:25.030078 30816 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 03:44:25.040587 30816 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 03:44:25.947985 30816 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 03:44:25.948079 30816 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms
I20260806 03:44:25.948099 30816 mini_ranger_kms.cc:227] Using postgres at 127.30.24.20:57701
2026-08-06 03:44:26,151 [I] DB FLAVOR :POSTGRES
2026-08-06 03:44:26,152 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:44:26,152 [I] Checking connection
2026-08-06 03:44:26,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 03:44:26,425 [I] connection success
2026-08-06 03:44:26,425 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 03:44:26,425 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 03:44:26,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 03:44:26,705 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 03:44:26,705 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:44:26,705 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 03:44:26,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.24.20:57701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 03:44:26,991 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 03:44:27.474460 30816 mini_ranger_kms.cc:310] Created kms service
I20260806 03:44:27.682013 30816 mini_ranger_kms.cc:326] Created kudu user
I20260806 03:44:27.726457 30816 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 03:44:27 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987865, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:28.117542 30816 mini_ranger_kms.cc:384] Added ranger policy
I20260806 03:44:28.117674 30816 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfG8t0l/build/release/bin/postgresql.jar:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfG8t0l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 03:44:28.117695 30816 mini_ranger_kms.cc:240] Using host: 127.30.24.20
I20260806 03:44:28.121845 30816 mini_ranger_kms.cc:276] Ranger KMS PID: 10795
I20260806 03:44:28.121927 30816 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.24.20:40529
Aug 06, 2026 3:44:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:44:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:44:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 3:44:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 3:44:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 3:44:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 03:44:30.431824 30816 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.402s user 0.000s sys 0.183s
I20260806 03:44:30.431912 30816 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 03:44:30.431950 30816 mini_ranger_kms.cc:401] 127.30.24.20:40529/kms/v1/keys
Aug 06 03:44:30 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785987870, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.24.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 03:44:31 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987870, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.24.20@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
Aug 06 03:44:31 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987870, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.24.20@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
Aug 06 03:44:33 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:44:33.870841 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33037
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:33037
--ranger_config_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:33.955027 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:33.955224 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:33.955246 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:33.956391 10848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:33.956436 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:33.956493 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:33.956527 10848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:33.956545 10848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:33.958102 10848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.24.20:40529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:33037
--ranger_config_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:33.958348 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:33.958573 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:33.963423 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:33.963438 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:33.963436 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:33.963729 10848 server_base.cc:1034] running on GCE node
I20260806 03:44:33.963967 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:33.964289 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:33.965444 10848 hybrid_clock.cc:648] HybridClock initialized: now 1785987873965428 us; error 54 us; skew 500 ppm
Aug 06 03:44:33 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987873, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:33.967986 10848 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:33.968428 10848 webserver.cc:460] Webserver started at http://127.30.24.62:39663/ using document root <none> and password file <none>
I20260806 03:44:33.968705 10848 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:33.968748 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:33.968896 10848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:44:33 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987873, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:34.023994 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "86431eed945c4532928d1c6ccfa50b73"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "484941f6691cbfbca7228ce13c5c81d6"
server_key_iv: "a8837b7a6f6d85e44cec70d8ee8d8edc"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:34.024521 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "86431eed945c4532928d1c6ccfa50b73"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "484941f6691cbfbca7228ce13c5c81d6"
server_key_iv: "a8837b7a6f6d85e44cec70d8ee8d8edc"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:34.026533 10848 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260806 03:44:34.039198 10863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:34.039439 10848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:34.039577 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "86431eed945c4532928d1c6ccfa50b73"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "484941f6691cbfbca7228ce13c5c81d6"
server_key_iv: "a8837b7a6f6d85e44cec70d8ee8d8edc"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:34.039681 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:34.063517 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:34.065975 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:34.066172 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:34.071992 10848 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:33037
I20260806 03:44:34.072023 10916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:33037 every 8 connection(s)
I20260806 03:44:34.072631 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:34.073407 10917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:34.075739 10917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Bootstrap starting.
I20260806 03:44:34.076368 10917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:34.076715 10917 log.cc:824] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:34.077216 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 10848
I20260806 03:44:34.077342 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:44:34.077535 30816 external_mini_cluster.cc:1428] Setting key 62636bdc433695968d08a6cb1676abfc
I20260806 03:44:34.077617 10917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: No bootstrap required, opened a new log
I20260806 03:44:34.080148 10917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:34.080329 10917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:34.080394 10917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86431eed945c4532928d1c6ccfa50b73, State: Initialized, Role: FOLLOWER
I20260806 03:44:34.080574 10917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:34.080680 10917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:44:34.080716 10917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:44:34.080793 10917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:34.081606 10917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:34.081739 10917 leader_election.cc:302] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86431eed945c4532928d1c6ccfa50b73; no voters:
I20260806 03:44:34.081947 10917 leader_election.cc:288] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:44:34.081979 10922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:34.082194 10922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 LEADER]: Becoming Leader. State: Replica: 86431eed945c4532928d1c6ccfa50b73, State: Running, Role: LEADER
I20260806 03:44:34.082340 10917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:34.082324 10922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:34.082762 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86431eed945c4532928d1c6ccfa50b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } } }
I20260806 03:44:34.082929 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:34.083189 10923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86431eed945c4532928d1c6ccfa50b73. Latest consensus state: current_term: 1 leader_uuid: "86431eed945c4532928d1c6ccfa50b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } } }
I20260806 03:44:34.083344 10923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:34.084009 10917 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 03:44:34.087877 10936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:34.088809 10936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 03:44:34 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:34.096118 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:34.078139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.30.24.62:33037)
0806 03:44:34.078368 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:34.078380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:34.078420 (+ 40us) server_negotiation.cc:407] Connection header received
0806 03:44:34.080167 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:34.080179 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:34.080259 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:34.080428 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:34.081411 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:34.083029 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:34.084249 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:34.087322 (+ 3073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:34.087507 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:34.090731 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:34.090738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:34.090739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:34.090753 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:34.093283 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:34.093774 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:34.093778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:34.093779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:34.093909 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:34.094151 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:34.094151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:34.094152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:34.094521 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:34.094690 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:34.095282 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 03:44:34.095434 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
I20260806 03:44:34.101068 10936 catalog_manager.cc:1347] Generated new cluster ID: 379b763e73fb4ed3bb8a267a3ab75918
I20260806 03:44:34.101130 10936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:44:34.119598 10936 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:44:34.120463 10936 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:44:34.126184 10936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Generated new TSK 0
I20260806 03:44:34.126454 10936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:44:34.864409 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:33037
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:34.967904 10962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:34.968098 10962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:34.968122 10962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:34.969372 10962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:34.969434 10962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:34.969478 10962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:44:34.971040 10962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.24.20:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:34.971295 10962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:34.971524 10962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:34.977568 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:34.977597 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:34.977589 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:34.977866 10962 server_base.cc:1034] running on GCE node
I20260806 03:44:34.978083 10962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:34.978374 10962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:34.979554 10962 hybrid_clock.cc:648] HybridClock initialized: now 1785987874979529 us; error 38 us; skew 500 ppm
Aug 06 03:44:34 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987874, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:34.982097 10962 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:44:34.982653 10962 webserver.cc:460] Webserver started at http://127.30.24.1:45963/ using document root <none> and password file <none>
I20260806 03:44:34.982914 10962 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:34.982983 10962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:34.983142 10962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:44:34 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987874, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:34.996032 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "2eab614389974e5ba66b569fc9623636"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "7c4f0cf66d4bf0871dc0d422833766fb"
server_key_iv: "f5dd5b69dc0f769053645cfe8e8471b9"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:34.996513 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "2eab614389974e5ba66b569fc9623636"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "7c4f0cf66d4bf0871dc0d422833766fb"
server_key_iv: "f5dd5b69dc0f769053645cfe8e8471b9"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.007728 10962 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.001s
I20260806 03:44:35.028056 10980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.028374 10962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:44:35.028522 10962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "2eab614389974e5ba66b569fc9623636"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "7c4f0cf66d4bf0871dc0d422833766fb"
server_key_iv: "f5dd5b69dc0f769053645cfe8e8471b9"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.028640 10962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:35.044921 10962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:35.047640 10962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:35.047895 10962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:35.048233 10962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:35.048720 10962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:35.048821 10962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.048938 10962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:35.049017 10962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.056639 10962 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:44673
I20260806 03:44:35.056761 11097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:44673 every 8 connection(s)
I20260806 03:44:35.057196 10962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:44:35.064932 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 10962
I20260806 03:44:35.065054 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:44:35.065232 30816 external_mini_cluster.cc:1428] Setting key 566526dc4761daad37eafe08a91d4cd1
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987874, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:35.073226 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:35.058049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33037 (local address 127.30.24.1:42739)
0806 03:44:35.062146 (+ 4097us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:35.062207 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:35.062856 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:35.063251 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:35.063256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:35.063519 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:35.063814 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:35.063820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.064892 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.064893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:35.066214 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:35.066218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.066792 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.066793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:35.066937 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:35.067584 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:35.067595 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:35.069347 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:35.071489 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:35.071496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:35.071499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:35.071824 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:35.072049 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:35.072050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:35.072050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:35.072183 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:35.072515 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:35.072522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:35.072711 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 03:44:35.072829 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 03:44:35.072990 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4052,"thread_start_us":87,"threads_started":1}
I20260806 03:44:35.073868 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:35.058082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:42739 (local address 127.30.24.62:33037)
0806 03:44:35.058227 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:35.058229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:35.063006 (+ 4777us) server_negotiation.cc:407] Connection header received
0806 03:44:35.063044 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:35.063045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:35.063061 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:35.063173 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:35.063934 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.064786 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.066317 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.066636 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.066708 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:35.069452 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:35.069453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:35.069454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:35.069461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:35.071363 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:35.071915 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:35.071916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:35.071916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:35.071983 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:35.072261 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:35.072262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:35.072262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:35.072399 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:35.072487 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:35.073669 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0806 03:44:35.073766 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260806 03:44:35.074606 11098 heartbeater.cc:347] Connected to a master server at 127.30.24.62:33037
I20260806 03:44:35.074720 11098 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:35.074908 11098 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:35.075477 10880 ts_manager.cc:194] Registered new tserver with Master: 2eab614389974e5ba66b569fc9623636 (127.30.24.1:44673)
I20260806 03:44:35.076339 10880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:42739
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:44:35.109200 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:33037
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 03:44:35.231092 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:35.231366 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:35.231402 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:35.233071 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:35.233173 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:35.233224 11104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:44:35.235383 11104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.24.20:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:35.235634 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:35.235986 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:35.242396 11116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:35.242558 11114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:35.242816 11112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:35.243163 11104 server_base.cc:1034] running on GCE node
I20260806 03:44:35.243400 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:35.243697 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:35.244866 11104 hybrid_clock.cc:648] HybridClock initialized: now 1785987875244814 us; error 68 us; skew 500 ppm
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:35.248297 11104 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:44:35.248924 11104 webserver.cc:460] Webserver started at http://127.30.24.2:37807/ using document root <none> and password file <none>
I20260806 03:44:35.249240 11104 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:35.249329 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:35.249477 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:35.262993 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "49924fb94b1a4f8ca512503fc6398373"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "55bfd4dce3eaf0e7350e68af00e09c84"
server_key_iv: "2a4484cffd953b148043bb219b21235a"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.263468 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "49924fb94b1a4f8ca512503fc6398373"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "55bfd4dce3eaf0e7350e68af00e09c84"
server_key_iv: "2a4484cffd953b148043bb219b21235a"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.268533 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260806 03:44:35.293823 11124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.294109 11104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 03:44:35.294214 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "49924fb94b1a4f8ca512503fc6398373"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "55bfd4dce3eaf0e7350e68af00e09c84"
server_key_iv: "2a4484cffd953b148043bb219b21235a"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.294314 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:35.327069 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:35.329619 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:35.329854 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:35.330179 11104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:35.330586 11104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:35.330679 11104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.330751 11104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:35.330809 11104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.337518 11104 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:39643
I20260806 03:44:35.338093 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:44:35.339095 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11104
I20260806 03:44:35.339184 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:44:35.339377 30816 external_mini_cluster.cc:1428] Setting key 7f95fef6c9c0dacd1f2442852acab6ae
I20260806 03:44:35.349433 11240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:39643 every 8 connection(s)
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:35.374728 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:35.353269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:45255 (local address 127.30.24.62:33037)
0806 03:44:35.353331 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:35.353333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:35.356795 (+ 3462us) server_negotiation.cc:407] Connection header received
0806 03:44:35.357690 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:35.357692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:35.357711 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:35.357775 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:35.363337 (+ 5562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.364134 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.365762 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.366095 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.366244 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:35.371266 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:35.371268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:35.371269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:35.371277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:35.373031 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:35.373534 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:35.373535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:35.373536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:35.373597 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:35.373911 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:35.373912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:35.373913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:35.374060 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:35.374116 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:35.374514 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 03:44:35.374612 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 03:44:35.375515 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:35.350288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33037 (local address 127.30.24.2:45255)
0806 03:44:35.356604 (+ 6316us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:35.356681 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:35.357553 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:35.362560 (+ 5007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:35.362567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:35.362863 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:35.363184 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:35.363191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.364278 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.364279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:35.365640 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:35.365643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.366339 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.366340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:35.366502 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:35.369403 (+ 2901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:35.369415 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:35.371140 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:35.373129 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:35.373137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:35.373139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:35.373438 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:35.373686 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:35.373686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:35.373687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:35.373825 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:35.374153 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:35.374159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:35.374344 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 03:44:35.375094 (+ 750us) client_negotiation.cc:240] Negotiation successful
0806 03:44:35.375272 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6252,"thread_start_us":122,"threads_started":1}
I20260806 03:44:35.376183 11241 heartbeater.cc:347] Connected to a master server at 127.30.24.62:33037
I20260806 03:44:35.376283 11241 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:35.376505 11241 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:35.376947 10880 ts_manager.cc:194] Registered new tserver with Master: 49924fb94b1a4f8ca512503fc6398373 (127.30.24.2:39643)
I20260806 03:44:35.377452 10880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:45255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:44:35.403421 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:33037
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:35.550028 11247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:35.550403 11247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:35.550514 11247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:35.552335 11247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:35.552481 11247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:35.552599 11247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:44:35.554872 11247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.24.20:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:35.555347 11247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:35.555713 11247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:35.562908 11255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:35.563251 11253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:35.563526 11252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:35.564905 11247 server_base.cc:1034] running on GCE node
I20260806 03:44:35.565151 11247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:35.565433 11247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:35.572809 11247 hybrid_clock.cc:648] HybridClock initialized: now 1785987875572781 us; error 40 us; skew 500 ppm
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:35.576179 11247 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:44:35.576773 11247 webserver.cc:460] Webserver started at http://127.30.24.3:44087/ using document root <none> and password file <none>
I20260806 03:44:35.577023 11247 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:35.577073 11247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:35.577200 11247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:35.592881 11247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "9265d9baed1e4a7ba9827242324caba4"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "d797fb0d5c35642bfc1c19f6df514ba1"
server_key_iv: "c8ab17d95b63a3594f06353e6452fecf"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.593478 11247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "9265d9baed1e4a7ba9827242324caba4"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "d797fb0d5c35642bfc1c19f6df514ba1"
server_key_iv: "c8ab17d95b63a3594f06353e6452fecf"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.596799 11247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260806 03:44:35.616642 11263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.616918 11247 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260806 03:44:35.617029 11247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "9265d9baed1e4a7ba9827242324caba4"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "d797fb0d5c35642bfc1c19f6df514ba1"
server_key_iv: "c8ab17d95b63a3594f06353e6452fecf"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.617136 11247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:35.663718 11247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:35.666276 11247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:35.666497 11247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:35.667166 11247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:35.671870 11247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:35.671929 11247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.671962 11247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:35.671984 11247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.678217 11247 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:41291
I20260806 03:44:35.678742 11247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:44:35.679231 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11247
I20260806 03:44:35.679320 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:44:35.679527 30816 external_mini_cluster.cc:1428] Setting key fdbdd127761f4e01d63633dcf57b618b
I20260806 03:44:35.687013 11378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:41291 every 8 connection(s)
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:35.700384 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:35.681504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:53067 (local address 127.30.24.62:33037)
0806 03:44:35.681570 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:35.681573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:35.681747 (+ 174us) server_negotiation.cc:407] Connection header received
0806 03:44:35.683088 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:35.683090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:35.683104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:35.683261 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:35.686829 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.687663 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.692265 (+ 4602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.692629 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.692678 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:35.696955 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:35.696957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:35.696957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:35.696966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:35.698711 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:35.699227 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:35.699228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:35.699229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:35.699290 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:35.699592 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:35.699593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:35.699593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:35.699736 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:35.699791 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:35.700191 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 03:44:35.700282 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 03:44:35.700930 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:35.681208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33037 (local address 127.30.24.3:53067)
0806 03:44:35.681611 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:35.681661 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:35.682944 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:35.683413 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:35.683418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:35.683696 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:35.684001 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:35.684008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.687796 (+ 3788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.687797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:35.689160 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:35.689163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:35.692798 (+ 3635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:35.692799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:35.692957 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:35.694624 (+ 1667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:35.694637 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:35.696439 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:35.698818 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:35.698826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:35.698829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:35.699132 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:35.699361 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:35.699361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:35.699362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:35.699497 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:35.699825 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:35.699831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:35.700025 (+ 194us) client_negotiation.cc:769] Sending connection context
0806 03:44:35.700509 (+ 484us) client_negotiation.cc:240] Negotiation successful
0806 03:44:35.700692 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":86,"threads_started":1}
I20260806 03:44:35.702085 11379 heartbeater.cc:347] Connected to a master server at 127.30.24.62:33037
I20260806 03:44:35.702186 11379 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:35.702376 11379 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:35.702800 10880 ts_manager.cc:194] Registered new tserver with Master: 9265d9baed1e4a7ba9827242324caba4 (127.30.24.3:41291)
I20260806 03:44:35.703285 10880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:53067
I20260806 03:44:35.712699 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:44:35.713487 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 10962
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:35.726156 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11104
I20260806 03:44:35.732666 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11247
I20260806 03:44:35.739848 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 10848
I20260806 03:44:35.759984 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33037
--webserver_interface=127.30.24.62
--webserver_port=39663
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:33037
--ranger_config_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:35.857052 11382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:35.857255 11382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:35.857280 11382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:35.858394 11382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:35.858438 11382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:35.858451 11382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:35.858464 11382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:35.858476 11382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:35.860045 11382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:33037
--ranger_config_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=39663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:35.860277 11382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:35.860564 11382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:35.865440 11387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:35.865477 11382 server_base.cc:1034] running on GCE node
W20260806 03:44:35.865625 11388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:35.865435 11390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:35.865995 11382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:35.866350 11382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:35.867525 11382 hybrid_clock.cc:648] HybridClock initialized: now 1785987875867498 us; error 39 us; skew 500 ppm
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:35.870605 11382 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:35.871150 11382 webserver.cc:460] Webserver started at http://127.30.24.62:39663/ using document root <none> and password file <none>
I20260806 03:44:35.871387 11382 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:35.871454 11382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:44:35 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:35.956041 11382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:35.957013 11398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:35.957240 11382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:44:35.957342 11382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "86431eed945c4532928d1c6ccfa50b73"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "484941f6691cbfbca7228ce13c5c81d6"
server_key_iv: "a8837b7a6f6d85e44cec70d8ee8d8edc"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:35.957818 11382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:35.988394 11382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:35.990928 11382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:35.991115 11382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:35.995914 11382 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:33037
I20260806 03:44:35.995955 11452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:33037 every 8 connection(s)
I20260806 03:44:35.996397 11382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:35.997795 11453 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:44:35.998464 11453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Bootstrap starting.
I20260806 03:44:35.999192 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11382
I20260806 03:44:35.999538 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:44673
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:33037
--webserver_port=45963
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:36.007246 11453 log.cc:824] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:36.009130 11453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:44:36.009433 11453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Bootstrap complete.
I20260806 03:44:36.011911 11453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:36.012174 11453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86431eed945c4532928d1c6ccfa50b73, State: Initialized, Role: FOLLOWER
I20260806 03:44:36.012384 11453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:36.012516 11453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:44:36.012581 11453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:44:36.012631 11453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:44:36.013638 11453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:36.013772 11453 leader_election.cc:302] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86431eed945c4532928d1c6ccfa50b73; no voters:
I20260806 03:44:36.013990 11453 leader_election.cc:288] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:44:36.014024 11459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:44:36.014149 11459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [term 2 LEADER]: Becoming Leader. State: Replica: 86431eed945c4532928d1c6ccfa50b73, State: Running, Role: LEADER
I20260806 03:44:36.014295 11453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:36.014324 11459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } }
I20260806 03:44:36.014643 11460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86431eed945c4532928d1c6ccfa50b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } } }
I20260806 03:44:36.014758 11460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:36.014957 11461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86431eed945c4532928d1c6ccfa50b73. Latest consensus state: current_term: 2 leader_uuid: "86431eed945c4532928d1c6ccfa50b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86431eed945c4532928d1c6ccfa50b73" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33037 } } }
I20260806 03:44:36.015040 11461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:36.015275 11465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:36.015887 11465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:44:36.016054 11465 catalog_manager.cc:1259] Loaded cluster ID: 379b763e73fb4ed3bb8a267a3ab75918
I20260806 03:44:36.016109 11465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:44:36.016422 11453 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 03:44:36.017630 11465 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:44:36.017829 11465 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Loaded TSK: 0
I20260806 03:44:36.018056 11465 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:44:36.132339 11455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:36.132819 11455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:36.132877 11455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:36.134749 11455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:36.134814 11455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:36.134855 11455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:44:36.137169 11455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:44673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=45963
--tserver_master_addrs=127.30.24.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:36.137420 11455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:36.137681 11455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:36.144136 11494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:36.144150 11496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:36.144285 11493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:36.144857 11455 server_base.cc:1034] running on GCE node
I20260806 03:44:36.145143 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:36.145447 11455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:36.146633 11455 hybrid_clock.cc:648] HybridClock initialized: now 1785987876146609 us; error 34 us; skew 500 ppm
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:36.150152 11455 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:44:36.150851 11455 webserver.cc:460] Webserver started at http://127.30.24.1:45963/ using document root <none> and password file <none>
I20260806 03:44:36.151125 11455 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:36.151207 11455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:36.170408 11455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:36.171262 11504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.171475 11455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:36.171571 11455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "2eab614389974e5ba66b569fc9623636"
format_stamp: "Formatted at 2026-08-06 03:44:34 on dist-test-slave-10pc"
server_key: "7c4f0cf66d4bf0871dc0d422833766fb"
server_key_iv: "f5dd5b69dc0f769053645cfe8e8471b9"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:36.172092 11455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:36.190887 11455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:36.193519 11455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:36.193709 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:36.193997 11455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:36.194375 11455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:36.194437 11455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.194485 11455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:36.194515 11455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.202123 11455 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:44673
I20260806 03:44:36.202224 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:44673 every 8 connection(s)
I20260806 03:44:36.202742 11455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:36.212612 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11455
I20260806 03:44:36.212921 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:39643
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:33037
--webserver_port=37807
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:36.216804 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.203876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:58255 (local address 127.30.24.62:33037)
0806 03:44:36.204165 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:36.204174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:36.204218 (+ 44us) server_negotiation.cc:407] Connection header received
0806 03:44:36.204922 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:36.204927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:36.204977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:36.205151 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:36.205927 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.207538 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.208762 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.209498 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.209634 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.212131 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:36.212136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:36.212137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:36.212150 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:36.214708 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.215264 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.215272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.215272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.215363 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.215735 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.215736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.215737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.216044 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:36.216140 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:36.216448 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 03:44:36.216600 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":119,"threads_started":1}
I20260806 03:44:36.216814 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.203710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33037 (local address 127.30.24.1:58255)
0806 03:44:36.203897 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:36.203945 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:36.204654 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:36.205197 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:36.205202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:36.205443 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:36.205783 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:36.205790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.207685 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.207687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:36.208660 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:36.208663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.209629 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.209630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:36.209736 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.210327 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:36.210336 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:36.212011 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:36.214821 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:36.214828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:36.214831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:36.215175 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:36.215463 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:36.215469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:36.215469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:36.215640 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:36.216117 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:36.216124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:36.216275 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 03:44:36.216390 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 03:44:36.216584 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260806 03:44:36.217731 11622 heartbeater.cc:347] Connected to a master server at 127.30.24.62:33037
I20260806 03:44:36.217852 11622 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.218040 11622 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
W20260806 03:44:36.218952 11622 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:33037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:44:36.219035 11622 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.219106 11622 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:36.219435 11622 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.219540 11622 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
W20260806 03:44:36.219837 11622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:44:36.346371 11626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:36.346705 11626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:36.346755 11626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:36.348608 11626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:36.348668 11626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:36.348712 11626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:44:36.351028 11626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:39643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=37807
--tserver_master_addrs=127.30.24.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:36.351320 11626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:36.351665 11626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:36.357262 11632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:36.357365 11634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:36.357270 11631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:36.358019 11626 server_base.cc:1034] running on GCE node
I20260806 03:44:36.358276 11626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:36.358531 11626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:36.359702 11626 hybrid_clock.cc:648] HybridClock initialized: now 1785987876359688 us; error 33 us; skew 500 ppm
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:36.363461 11626 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:44:36.364001 11626 webserver.cc:460] Webserver started at http://127.30.24.2:37807/ using document root <none> and password file <none>
I20260806 03:44:36.364262 11626 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:36.364337 11626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:36.386294 11626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:36.387151 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.387367 11626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:36.387467 11626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "49924fb94b1a4f8ca512503fc6398373"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "55bfd4dce3eaf0e7350e68af00e09c84"
server_key_iv: "2a4484cffd953b148043bb219b21235a"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:36.387946 11626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:36.402019 11626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:36.404580 11626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:36.404762 11626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:36.405056 11626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:36.405470 11626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:36.405530 11626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.405565 11626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:36.405593 11626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.414058 11626 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:39643
I20260806 03:44:36.414116 11758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:39643 every 8 connection(s)
I20260806 03:44:36.414536 11626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:36.423254 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11626
I20260806 03:44:36.423624 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.24.20:40529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:41291
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:33037
--webserver_port=44087
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:36.425904 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.415488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48809 (local address 127.30.24.62:33037)
0806 03:44:36.415569 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:36.415572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:36.415859 (+ 287us) server_negotiation.cc:407] Connection header received
0806 03:44:36.416392 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:36.416394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:36.416408 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:36.416557 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:36.417202 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.418089 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.419204 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.419552 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.419645 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.422123 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:36.422125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:36.422125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:36.422133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:36.424112 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.424659 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.424660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.424660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.424728 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.425075 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.425076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.425077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.425230 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:36.425282 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:36.425704 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 03:44:36.425802 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 03:44:36.426115 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.415493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33037 (local address 127.30.24.2:48809)
0806 03:44:36.415694 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:36.415741 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:36.416274 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:36.416650 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:36.416655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:36.416876 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:36.417089 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:36.417094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.418211 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.418213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:36.419102 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:36.419104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.419663 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.419664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:36.419767 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.420353 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:36.420362 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:36.422016 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:36.424211 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:36.424218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:36.424223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:36.424542 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:36.424814 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:36.424815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:36.424815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:36.424956 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:36.425341 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:36.425348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:36.425540 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 03:44:36.425678 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 03:44:36.425839 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
I20260806 03:44:36.426951 11759 heartbeater.cc:347] Connected to a master server at 127.30.24.62:33037
I20260806 03:44:36.427081 11759 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.427280 11759 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
W20260806 03:44:36.428059 11759 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:33037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:44:36.428153 11759 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.428223 11759 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:36.428679 11759 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.428764 11759 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
W20260806 03:44:36.429179 11759 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:44:36.562253 11762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:36.562585 11762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:36.562636 11762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:36.564607 11762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:36.564677 11762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:36.564726 11762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:44:36.566650 11762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.24.20:40529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:41291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=44087
--tserver_master_addrs=127.30.24.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:36.566957 11762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:36.567235 11762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:36.572757 11767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:36.573167 11768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:36.573407 11770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:36.574221 11762 server_base.cc:1034] running on GCE node
I20260806 03:44:36.574450 11762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:36.574745 11762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:36.575906 11762 hybrid_clock.cc:648] HybridClock initialized: now 1785987876575885 us; error 38 us; skew 500 ppm
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:36.578974 11762 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:44:36.579559 11762 webserver.cc:460] Webserver started at http://127.30.24.3:44087/ using document root <none> and password file <none>
I20260806 03:44:36.579892 11762 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:36.579981 11762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:36.595775 11762 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:36.596539 11779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.596722 11762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:36.596819 11762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "9265d9baed1e4a7ba9827242324caba4"
format_stamp: "Formatted at 2026-08-06 03:44:35 on dist-test-slave-10pc"
server_key: "d797fb0d5c35642bfc1c19f6df514ba1"
server_key_iv: "c8ab17d95b63a3594f06353e6452fecf"
server_key_version: "kuduclusterkey@0"
I20260806 03:44:36.597219 11762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:36.611558 11762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:36.614377 11762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:36.614568 11762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:36.614883 11762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:36.615285 11762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:36.615338 11762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.615372 11762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:36.615399 11762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:36.623008 11762 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:41291
I20260806 03:44:36.623102 11897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:41291 every 8 connection(s)
I20260806 03:44:36.623539 11762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:44:36.626423 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11762
Aug 06 03:44:36 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:36.635244 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.626873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.30.24.62:33037)
0806 03:44:36.626985 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:36.626987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:36.627214 (+ 227us) server_negotiation.cc:407] Connection header received
0806 03:44:36.627255 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:36.627257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:36.627270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:36.627355 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:36.628079 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.628638 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.629431 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.629735 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.629800 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.631430 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:36.631432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:36.631432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:36.631441 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:36.633953 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.634365 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.634366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.634367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.634423 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.634640 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.634641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.634642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.634797 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:36.634960 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:36.635013 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 03:44:36.635101 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":56,"threads_started":1}
I20260806 03:44:36.635787 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.624923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:33769 (local address 127.30.24.62:33037)
0806 03:44:36.624974 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:36.624977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:36.625219 (+ 242us) server_negotiation.cc:407] Connection header received
0806 03:44:36.625789 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:36.625791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:36.625805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:36.625857 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:36.627040 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.627881 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.629495 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.629801 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.629856 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.632335 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:36.632336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:36.632337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:36.632344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:36.634194 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.634699 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.634699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.634700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.634744 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:36.635014 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:36.635015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:36.635016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:36.635127 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:36.635183 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:36.635621 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 03:44:36.635695 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":23}
I20260806 03:44:36.636003 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:36.624908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:33037 (local address 127.30.24.3:33769)
0806 03:44:36.625088 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:36.625125 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:36.625647 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:36.626325 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:36.626330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:36.626668 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:36.626955 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:36.626961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.628000 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.628002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:36.629403 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:36.629406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:36.629925 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:36.629927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:36.630106 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:36.630800 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:36.630809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:36.632234 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:36.634268 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:36.634275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:36.634278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:36.634592 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:36.634811 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:36.634812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:36.634813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:36.634940 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:36.635210 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:36.635216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:36.635450 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 03:44:36.635598 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 03:44:36.635768 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260806 03:44:36.636834 11898 heartbeater.cc:347] Connected to a master server at 127.30.24.62:33037
I20260806 03:44:36.636965 11898 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.637159 11898 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
W20260806 03:44:36.638617 11898 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:33037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:44:36.638707 11898 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.638780 11898 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:36.639067 11898 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:36.639140 11898 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
W20260806 03:44:36.639513 11898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 03:44:37 dist-test-slave-10pc krb5kdc[10392](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785987877, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:37.220037 11622 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:37.220218 11622 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:37.220695 11417 ts_manager.cc:194] Registered new tserver with Master: 2eab614389974e5ba66b569fc9623636 (127.30.24.1:44673)
I20260806 03:44:37.221870 11417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:58255
I20260806 03:44:37.429381 11759 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:37.429565 11759 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:37.429937 11417 ts_manager.cc:194] Registered new tserver with Master: 49924fb94b1a4f8ca512503fc6398373 (127.30.24.2:39643)
I20260806 03:44:37.430576 11417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:48809
Aug 06 03:44:37 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987877, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:37.639719 11898 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:37.639917 11898 heartbeater.cc:511] Master 127.30.24.62:33037 requested a full tablet report, sending...
I20260806 03:44:37.640314 11417 ts_manager.cc:194] Registered new tserver with Master: 9265d9baed1e4a7ba9827242324caba4 (127.30.24.3:41291)
I20260806 03:44:37.640792 11417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:33769
I20260806 03:44:37.644791 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:44:37.652552 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.645825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.30.24.62:33037)
0806 03:44:37.645940 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.645943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.646066 (+ 123us) server_negotiation.cc:407] Connection header received
0806 03:44:37.646138 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.646139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.646152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.646224 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:37.646795 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.647314 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.648059 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.648291 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.648368 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.649704 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:37.649705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:37.649705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:37.649712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:37.651280 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:37.651667 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:37.651668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:37.651669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:37.651735 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:37.651945 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:37.651946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:37.651946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:37.652092 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:37.652205 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.652316 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.652407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260806 03:44:37.654593 11417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:44:37.656324 11417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:44:37.664665 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.659435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.30.24.1:44673)
0806 03:44:37.660140 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.660145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.660236 (+ 91us) server_negotiation.cc:407] Connection header received
0806 03:44:37.660338 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.660340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.660355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.660405 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:37.660667 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.661786 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.663280 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.664145 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.664206 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.664271 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.664523 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.664575 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":50,"threads_started":1}
I20260806 03:44:37.664772 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.659111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:41291 (local address 127.0.0.1:59662)
0806 03:44:37.659954 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:37.659966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:37.660029 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:37.660264 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:37.660267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:37.660270 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:37.660343 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.660345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.661643 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.661644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.663237 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.663241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.664204 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.664205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.664285 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.664300 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 03:44:37.664349 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 03:44:37.664693 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":83,"threads_started":1}
I20260806 03:44:37.665220 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.659353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.30.24.3:41291)
0806 03:44:37.659515 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.659523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.660073 (+ 550us) server_negotiation.cc:407] Connection header received
0806 03:44:37.660113 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.660115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.660131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.660416 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:37.660498 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.661530 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.663341 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.664073 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.664957 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.665065 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.665092 (+ 27us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.665131 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
I20260806 03:44:37.665432 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.659308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44673 (local address 127.0.0.1:45522)
0806 03:44:37.660187 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:37.660196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:37.660259 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:37.660543 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:37.660544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:37.660546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:37.660591 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.660593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.661891 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.661892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.663174 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.663178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.664335 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.664336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.664403 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.664407 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:44:37.665219 (+ 812us) client_negotiation.cc:240] Negotiation successful
0806 03:44:37.665264 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":79,"threads_started":1}
I20260806 03:44:37.666036 11828 tablet_service.cc:1467] Processing CreateTablet for tablet 68296fa8f3de49dd92ac92c64eb2ee91 (DEFAULT_TABLE table=test-table [id=fd74f5a90af04a4b915d9efe9cafebbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:37.666462 11828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68296fa8f3de49dd92ac92c64eb2ee91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:37.670145 11932 tablet_bootstrap.cc:492] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Bootstrap starting.
I20260806 03:44:37.670985 11932 tablet_bootstrap.cc:654] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:37.671154 11534 tablet_service.cc:1467] Processing CreateTablet for tablet 68296fa8f3de49dd92ac92c64eb2ee91 (DEFAULT_TABLE table=test-table [id=fd74f5a90af04a4b915d9efe9cafebbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:37.671336 11932 log.cc:824] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:37.671587 11534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68296fa8f3de49dd92ac92c64eb2ee91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:37.676000 11934 tablet_bootstrap.cc:492] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: Bootstrap starting.
I20260806 03:44:37.676242 11932 tablet_bootstrap.cc:492] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: No bootstrap required, opened a new log
I20260806 03:44:37.676401 11932 ts_tablet_manager.cc:1397] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 03:44:37.676827 11934 tablet_bootstrap.cc:654] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:37.677189 11934 log.cc:824] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:37.678156 11934 tablet_bootstrap.cc:492] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: No bootstrap required, opened a new log
I20260806 03:44:37.678228 11934 ts_tablet_manager.cc:1397] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:44:37.679903 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.659075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:39643 (local address 127.0.0.1:46092)
0806 03:44:37.659457 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:37.659501 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:37.659566 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:37.672563 (+ 12997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:37.672565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:37.672568 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:37.672623 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.672625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.674009 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.674011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.675059 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.675063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.679669 (+ 4606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.679670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.679743 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.679749 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:44:37.679781 (+ 32us) client_negotiation.cc:240] Negotiation successful
0806 03:44:37.679825 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":88,"threads_started":1}
I20260806 03:44:37.680812 11934 raft_consensus.cc:348] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.681006 11934 raft_consensus.cc:374] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:37.681046 11934 raft_consensus.cc:729] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eab614389974e5ba66b569fc9623636, State: Initialized, Role: FOLLOWER
I20260806 03:44:37.681156 11934 consensus_queue.cc:260] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.681290 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.659175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.30.24.2:39643)
0806 03:44:37.659692 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.659696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.659704 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:44:37.659733 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.659735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.659771 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.659844 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:37.672772 (+ 12928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.673892 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.675186 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.678672 (+ 3486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.680251 (+ 1579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.680328 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.680357 (+ 29us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.680401 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":58,"threads_started":1}
I20260806 03:44:37.681408 11934 ts_tablet_manager.cc:1428] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260806 03:44:37.681529 11622 heartbeater.cc:503] Master 127.30.24.62:33037 was elected leader, sending a full tablet report...
I20260806 03:44:37.681775 11932 raft_consensus.cc:348] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.681936 11932 raft_consensus.cc:374] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:37.681967 11932 raft_consensus.cc:729] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9265d9baed1e4a7ba9827242324caba4, State: Initialized, Role: FOLLOWER
I20260806 03:44:37.682093 11932 consensus_queue.cc:260] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.682340 11932 ts_tablet_manager.cc:1428] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 03:44:37.682248 11691 tablet_service.cc:1467] Processing CreateTablet for tablet 68296fa8f3de49dd92ac92c64eb2ee91 (DEFAULT_TABLE table=test-table [id=fd74f5a90af04a4b915d9efe9cafebbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:37.682461 11898 heartbeater.cc:503] Master 127.30.24.62:33037 was elected leader, sending a full tablet report...
I20260806 03:44:37.682591 11691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68296fa8f3de49dd92ac92c64eb2ee91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:37.685863 11938 tablet_bootstrap.cc:492] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Bootstrap starting.
I20260806 03:44:37.686594 11938 tablet_bootstrap.cc:654] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:37.686916 11938 log.cc:824] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:37.687814 11938 tablet_bootstrap.cc:492] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: No bootstrap required, opened a new log
I20260806 03:44:37.687877 11938 ts_tablet_manager.cc:1397] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:44:37.690055 11938 raft_consensus.cc:348] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.690207 11938 raft_consensus.cc:374] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:37.690235 11938 raft_consensus.cc:729] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49924fb94b1a4f8ca512503fc6398373, State: Initialized, Role: FOLLOWER
I20260806 03:44:37.690368 11938 consensus_queue.cc:260] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.690609 11938 ts_tablet_manager.cc:1428] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:44:37.690681 11759 heartbeater.cc:503] Master 127.30.24.62:33037 was elected leader, sending a full tablet report...
W20260806 03:44:37.705618 11623 tablet.cc:2369] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 03:44:37 dist-test-slave-10pc krb5kdc[10392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987877, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for HTTP/127.30.24.20@KRBTEST.COM
I20260806 03:44:37.813822 11937 raft_consensus.cc:482] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:37.813982 11937 raft_consensus.cc:504] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.814385 11937 leader_election.cc:288] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49924fb94b1a4f8ca512503fc6398373 (127.30.24.2:39643), 2eab614389974e5ba66b569fc9623636 (127.30.24.1:44673)
I20260806 03:44:37.820392 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.815122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:52219 (local address 127.30.24.2:39643)
0806 03:44:37.815215 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.815217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.816311 (+ 1094us) server_negotiation.cc:407] Connection header received
0806 03:44:37.816430 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.816431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.816441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.816497 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:37.816791 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.817434 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.819250 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.819805 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.819854 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.819863 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.820277 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.820324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 03:44:37.820894 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.814647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:39643 (local address 127.30.24.3:52219)
0806 03:44:37.816224 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:37.816237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:37.816347 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:37.816580 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:37.816581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:37.816584 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:37.816698 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.816701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.817762 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.817763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.819140 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.819143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.820104 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.820105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.820174 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.820180 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:44:37.820735 (+ 555us) client_negotiation.cc:240] Negotiation successful
0806 03:44:37.820784 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":110,"threads_started":1}
I20260806 03:44:37.821394 11712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68296fa8f3de49dd92ac92c64eb2ee91" candidate_uuid: "9265d9baed1e4a7ba9827242324caba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49924fb94b1a4f8ca512503fc6398373" is_pre_election: true
I20260806 03:44:37.821578 11712 raft_consensus.cc:2454] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9265d9baed1e4a7ba9827242324caba4 in term 0.
I20260806 03:44:37.821807 11781 leader_election.cc:302] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49924fb94b1a4f8ca512503fc6398373, 9265d9baed1e4a7ba9827242324caba4; no voters:
I20260806 03:44:37.821951 11937 raft_consensus.cc:2781] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:44:37.822057 11937 raft_consensus.cc:482] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:37.822086 11937 raft_consensus.cc:3037] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:37.822980 11937 raft_consensus.cc:504] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.823127 11937 leader_election.cc:288] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [CANDIDATE]: Term 1 election: Requested vote from peers 49924fb94b1a4f8ca512503fc6398373 (127.30.24.2:39643), 2eab614389974e5ba66b569fc9623636 (127.30.24.1:44673)
I20260806 03:44:37.823313 11712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68296fa8f3de49dd92ac92c64eb2ee91" candidate_uuid: "9265d9baed1e4a7ba9827242324caba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49924fb94b1a4f8ca512503fc6398373"
I20260806 03:44:37.823405 11712 raft_consensus.cc:3037] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:37.824265 11712 raft_consensus.cc:2454] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9265d9baed1e4a7ba9827242324caba4 in term 1.
I20260806 03:44:37.825318 11781 leader_election.cc:302] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49924fb94b1a4f8ca512503fc6398373, 9265d9baed1e4a7ba9827242324caba4; no voters:
I20260806 03:44:37.825415 11937 raft_consensus.cc:2781] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:37.825579 11937 raft_consensus.cc:686] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 1 LEADER]: Becoming Leader. State: Replica: 9265d9baed1e4a7ba9827242324caba4, State: Running, Role: LEADER
I20260806 03:44:37.825702 11937 consensus_queue.cc:237] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } }
I20260806 03:44:37.826589 11417 catalog_manager.cc:5495] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9265d9baed1e4a7ba9827242324caba4 (127.30.24.3). New cstate: current_term: 1 leader_uuid: "9265d9baed1e4a7ba9827242324caba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9265d9baed1e4a7ba9827242324caba4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20260806 03:44:37.828191 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.814851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44673 (local address 127.30.24.3:37005)
0806 03:44:37.821914 (+ 7063us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:37.821927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:37.822001 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:37.822196 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:37.822197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:37.822198 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:37.822228 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.822229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.823037 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.823038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.827182 (+ 4144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:37.827185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.827975 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.827975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:37.828017 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.828020 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:44:37.828088 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 03:44:37.828126 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7049,"thread_start_us":59,"threads_started":1}
I20260806 03:44:37.828208 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.815172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:37005 (local address 127.30.24.1:44673)
0806 03:44:37.815247 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.815249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.821973 (+ 6724us) server_negotiation.cc:407] Connection header received
0806 03:44:37.822080 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.822081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.822095 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.822128 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:37.822311 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.822795 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.827354 (+ 4559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.827881 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.827919 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.827928 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.828094 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.828136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 03:44:37.828512 11574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68296fa8f3de49dd92ac92c64eb2ee91" candidate_uuid: "9265d9baed1e4a7ba9827242324caba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eab614389974e5ba66b569fc9623636" is_pre_election: true
I20260806 03:44:37.828679 11574 raft_consensus.cc:2454] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9265d9baed1e4a7ba9827242324caba4 in term 0.
I20260806 03:44:37.828806 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68296fa8f3de49dd92ac92c64eb2ee91" candidate_uuid: "9265d9baed1e4a7ba9827242324caba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eab614389974e5ba66b569fc9623636"
I20260806 03:44:37.828866 11573 raft_consensus.cc:3037] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:37.829756 11573 raft_consensus.cc:2454] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9265d9baed1e4a7ba9827242324caba4 in term 1.
I20260806 03:44:37.849313 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.845905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.30.24.3:41291)
0806 03:44:37.845956 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:37.845958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:37.845966 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:44:37.846008 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:37.846009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:37.846033 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:37.846116 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:44:37.846285 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.847030 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.848192 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:37.848492 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:37.848554 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:37.848774 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:44:37.848910 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:44:37.848993 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:37.849174 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 03:44:37.849199 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:44:37.851828 11573 raft_consensus.cc:1261] T 68296fa8f3de49dd92ac92c64eb2ee91 P 2eab614389974e5ba66b569fc9623636 [term 1 FOLLOWER]: Refusing update from remote peer 9265d9baed1e4a7ba9827242324caba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:37.852205 11937 consensus_queue.cc:1035] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eab614389974e5ba66b569fc9623636" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:37.854884 11712 raft_consensus.cc:1261] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 1 FOLLOWER]: Refusing update from remote peer 9265d9baed1e4a7ba9827242324caba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:37.855160 11937 consensus_queue.cc:1035] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49924fb94b1a4f8ca512503fc6398373" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:37.859798 11948 mvcc.cc:205] Tried to move back new op lower bound from 7315406347678445568 to 7315406347574591488. Current Snapshot: MvccSnapshot[applied={T|T < 7315406347678445568}]
I20260806 03:44:37.860888 11949 mvcc.cc:205] Tried to move back new op lower bound from 7315406347678445568 to 7315406347574591488. Current Snapshot: MvccSnapshot[applied={T|T < 7315406347678445568}]
I20260806 03:44:37.863847 11416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:44:37.863955 11416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:44:37.864769 11416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86431eed945c4532928d1c6ccfa50b73: Sending DeleteTablet for 3 replicas of tablet 68296fa8f3de49dd92ac92c64eb2ee91
I20260806 03:44:37.865331 11691 tablet_service.cc:1514] Processing DeleteTablet for tablet 68296fa8f3de49dd92ac92c64eb2ee91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:37 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:46092
I20260806 03:44:37.865676 11534 tablet_service.cc:1514] Processing DeleteTablet for tablet 68296fa8f3de49dd92ac92c64eb2ee91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:37 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:45522
I20260806 03:44:37.866098 11828 tablet_service.cc:1514] Processing DeleteTablet for tablet 68296fa8f3de49dd92ac92c64eb2ee91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:37 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59662
I20260806 03:44:37.866592 11956 tablet_replica.cc:330] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: stopping tablet replica
I20260806 03:44:37.866710 11956 raft_consensus.cc:2229] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:44:37.866712 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11455
I20260806 03:44:37.866806 11956 pending_rounds.cc:62] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Trying to abort 2 pending ops.
I20260806 03:44:37.866845 11956 pending_rounds.cc:68] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7315406347678445568 op_type: WRITE_OP write_request { tablet_id: "68296fa8f3de49dd92ac92c64eb2ee91" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\321\210\320\323\006\"8\n\ntest-admin\022*\n fd74f5a90af04a4b915d9efe9cafebbe\020\001\030\001 \001(\001" signature: "<redacted>""!\360\350\236\026\202WUm\3350l}$\255\246\253\226\213\223\370\226Fp\246\251\210\013\246T80\326m\333\347~$\310\315F\363Ql\000\177\347\007\327\301\276\007\007\3314I5\373\377\206\240\242\007\262" signing_key_seq_num: 0 } } request_id { client_id: "1707b69a4f854d9384a028750c7e987a" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260806 03:44:37.867076 11956 pending_rounds.cc:68] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7315406347574591488 op_type: NO_OP noop_request { }
I20260806 03:44:37.867115 11956 raft_consensus.cc:2866] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260806 03:44:37.867304 11958 tablet_replica.cc:330] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: stopping tablet replica
I20260806 03:44:37.867390 11958 raft_consensus.cc:2229] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:44:37.867477 11958 raft_consensus.cc:2258] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 03:44:37.871933 11952 consensus_queue.cc:1159] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "49924fb94b1a4f8ca512503fc6398373" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260806 03:44:37.872144 11956 raft_consensus.cc:2258] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:37.872748 11958 ts_tablet_manager.cc:1905] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:37.875484 11958 ts_tablet_manager.cc:1918] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:37.875623 11958 log.cc:1196] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/68296fa8f3de49dd92ac92c64eb2ee91
I20260806 03:44:37.875934 11956 ts_tablet_manager.cc:1905] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:37.876585 11958 ts_tablet_manager.cc:1939] T 68296fa8f3de49dd92ac92c64eb2ee91 P 9265d9baed1e4a7ba9827242324caba4: Deleting consensus metadata
I20260806 03:44:37.877619 11956 ts_tablet_manager.cc:1918] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:37.877691 11956 log.cc:1196] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/68296fa8f3de49dd92ac92c64eb2ee91
I20260806 03:44:37.877737 11402 catalog_manager.cc:4841] TS 9265d9baed1e4a7ba9827242324caba4 (127.30.24.3:41291): tablet 68296fa8f3de49dd92ac92c64eb2ee91 (table test-table [id=fd74f5a90af04a4b915d9efe9cafebbe]) successfully deleted
I20260806 03:44:37.877951 11956 ts_tablet_manager.cc:1939] T 68296fa8f3de49dd92ac92c64eb2ee91 P 49924fb94b1a4f8ca512503fc6398373: Deleting consensus metadata
I20260806 03:44:37.878468 11400 catalog_manager.cc:4841] TS 49924fb94b1a4f8ca512503fc6398373 (127.30.24.2:39643): tablet 68296fa8f3de49dd92ac92c64eb2ee91 (table test-table [id=fd74f5a90af04a4b915d9efe9cafebbe]) successfully deleted
W20260806 03:44:37.896783 11400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:44673 (error 108)
I20260806 03:44:37.897759 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11626
I20260806 03:44:37.897775 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:37.897612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:44673 (local address 127.0.0.1:45524)
0806 03:44:37.897671 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:37.897735 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 03:44:37.900384 11400 catalog_manager.cc:4584] TS 2eab614389974e5ba66b569fc9623636 (127.30.24.1:44673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68296fa8f3de49dd92ac92c64eb2ee91: Network error: Client connection negotiation failed: client connection to 127.30.24.1:44673: connect: Connection refused (error 111)
I20260806 03:44:37.905979 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11762
I20260806 03:44:37.913554 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11382
I20260806 03:44:37.922092 30816 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 03:44:38.266824 30816 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 03:44:38.267113 30816 mini_ranger.cc:66] Stopping Ranger...
I20260806 03:44:38.622488 30816 mini_ranger.cc:68] Stopped Ranger
2026-08-06 03:44:38.622 UTC [10424] LOG: received smart shutdown request
2026-08-06 03:44:38.626 UTC [10424] LOG: background worker "logical replication launcher" (PID 10433) exited with exit code 1
2026-08-06 03:44:38.626 UTC [10428] LOG: shutting down
2026-08-06 03:44:38.810 UTC [10424] LOG: database system is shut down
2026-08-06T03:44:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48630 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:44:38 dist-test-slave-10pc krb5kdc[11968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:44:38 dist-test-slave-10pc krb5kdc[11968](info): set up 2 sockets
Aug 06 03:44:38 dist-test-slave-10pc krb5kdc[11968](info): commencing operation
krb5kdc: starting...
W20260806 03:44:39.985556 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:44:40.111801 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38975
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38975,127.30.24.61:44033,127.30.24.60:46097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:40.196272 11982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:40.196506 11982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:40.196529 11982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:40.197597 11982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:40.197633 11982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:40.197646 11982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:40.197659 11982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:40.197670 11982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:40.199056 11982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38975,127.30.24.61:44033,127.30.24.60:46097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:40.199249 11982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:40.199415 11982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:40.204319 11988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:40.204319 11990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:40.204404 11987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:40.204433 11982 server_base.cc:1034] running on GCE node
I20260806 03:44:40.204779 11982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:40.205072 11982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:40.206244 11982 hybrid_clock.cc:648] HybridClock initialized: now 1785987880206229 us; error 26 us; skew 500 ppm
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:40.211690 11982 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:40.212209 11982 webserver.cc:460] Webserver started at http://127.30.24.62:35403/ using document root <none> and password file <none>
I20260806 03:44:40.212440 11982 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:40.212556 11982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:40.212715 11982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:40.213794 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "8f3394ba9548400789d95b9b972b87bf"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "883c456bb182d003a385b41b34c8a76f"
server_key_iv: "662eb35d1800ff8acf3d17dd7eb7bc74"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.214180 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "8f3394ba9548400789d95b9b972b87bf"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "883c456bb182d003a385b41b34c8a76f"
server_key_iv: "662eb35d1800ff8acf3d17dd7eb7bc74"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.215549 11982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:44:40.216257 11997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:40.216483 11982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:44:40.216598 11982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "8f3394ba9548400789d95b9b972b87bf"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "883c456bb182d003a385b41b34c8a76f"
server_key_iv: "662eb35d1800ff8acf3d17dd7eb7bc74"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.216682 11982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:40.249119 11982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:40.251221 11982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:40.251394 11982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:40.255841 11982 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:38975
I20260806 03:44:40.255884 12052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:38975 every 8 connection(s)
I20260806 03:44:40.256282 11982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:40.257210 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:40.257791 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 11982
I20260806 03:44:40.257916 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:44:40.258104 30816 external_mini_cluster.cc:1428] Setting key a2166f419ba8fa2989af9e311ee28d45
I20260806 03:44:40.258912 12053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.24.61@KRBTEST.COM; defaulting to no policy
I20260806 03:44:40.270241 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.259889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.30.24.62:38975)
0806 03:44:40.260019 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.260024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.260067 (+ 43us) server_negotiation.cc:407] Connection header received
0806 03:44:40.260788 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.260791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.260808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.260885 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.261600 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.262964 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.263836 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.264388 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.264548 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.266811 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.266820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.266820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.266828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.268490 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.268866 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.268869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.268870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.268958 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.269190 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.269191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.269192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.269570 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.269640 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.269916 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.270088 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260806 03:44:40.270268 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.259674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38975 (local address 127.0.0.1:55168)
0806 03:44:40.259872 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.259924 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.260637 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.260929 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.260932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.261317 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.261485 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.261490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.263054 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.263055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.263747 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.263749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.264529 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.264530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.264591 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.265163 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.265172 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.266715 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.268569 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.268575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.268577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.268769 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.269027 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.269027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.269027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.269107 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.269673 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.269680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.269776 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.269876 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.270012 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.61@KRBTEST.COM" created.
I20260806 03:44:40.271102 12053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:44:40.271472 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.271347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51470)
0806 03:44:40.271399 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.271445 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 03:44:40.271637 12001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
I20260806 03:44:40.272076 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.271986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51482)
0806 03:44:40.272029 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.272055 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 03:44:40.272209 12053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:44033: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab.
Entry for principal kudu/127.30.24.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab.
I20260806 03:44:40.287541 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:44033
--webserver_interface=127.30.24.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab
--principal=kudu/127.30.24.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38975,127.30.24.61:44033,127.30.24.60:46097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:40.321413 12053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } attempt: 1
I20260806 03:44:40.321894 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.321761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51496)
0806 03:44:40.321823 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.321865 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 03:44:40.322261 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.322146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51512)
0806 03:44:40.322206 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.322230 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 03:44:40.322350 12053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:44033: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
W20260806 03:44:40.370621 12060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:40.370870 12060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:40.370908 12060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:40.372048 12060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:40.372114 12060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:40.372138 12060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:40.372157 12060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:40.372190 12060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:40.373705 12060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38975,127.30.24.61:44033,127.30.24.60:46097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab
--principal=kudu/127.30.24.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.24.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:40.373980 12060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:40.374214 12060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:40.379041 12065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:40.379060 12068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:40.379114 12060 server_base.cc:1034] running on GCE node
W20260806 03:44:40.379056 12066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:40.379444 12060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:40.379724 12060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:40.380873 12060 hybrid_clock.cc:648] HybridClock initialized: now 1785987880380844 us; error 62 us; skew 500 ppm
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:40.383364 12060 init.cc:377] Logged in from keytab as kudu/127.30.24.61@KRBTEST.COM (short username kudu)
I20260806 03:44:40.383845 12060 webserver.cc:460] Webserver started at http://127.30.24.61:42669/ using document root <none> and password file <none>
I20260806 03:44:40.384092 12060 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:40.384159 12060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:40.384315 12060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:40.385435 12060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data/instance:
uuid: "be0bbb92d1504e35b295724665c5f371"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "de2e0de2f4d789e219f74d3a55869aee"
server_key_iv: "bcea1e38fdb49b2d4a83bb22786d3f10"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.385807 12060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal/instance:
uuid: "be0bbb92d1504e35b295724665c5f371"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "de2e0de2f4d789e219f74d3a55869aee"
server_key_iv: "bcea1e38fdb49b2d4a83bb22786d3f10"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.387115 12060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:44:40.387852 12075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:40.388025 12060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:44:40.388123 12060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal
uuid: "be0bbb92d1504e35b295724665c5f371"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "de2e0de2f4d789e219f74d3a55869aee"
server_key_iv: "bcea1e38fdb49b2d4a83bb22786d3f10"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.388206 12060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:40.390481 12053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } attempt: 2
I20260806 03:44:40.390872 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.390742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51520)
0806 03:44:40.390801 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.390832 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 03:44:40.391175 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.391082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51536)
0806 03:44:40.391127 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.391151 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 03:44:40.391266 12053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:44033: Network error: Client connection negotiation failed: client connection to 127.30.24.61:44033: connect: Connection refused (error 111)
I20260806 03:44:40.403921 12060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:40.406301 12060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:40.406461 12060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:40.410573 12060 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.61:44033
I20260806 03:44:40.410625 12128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.61:44033 every 8 connection(s)
I20260806 03:44:40.410985 12060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
I20260806 03:44:40.411824 12129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:40.412700 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12060
I20260806 03:44:40.412842 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-1/wal/instance
I20260806 03:44:40.413031 30816 external_mini_cluster.cc:1428] Setting key f40427c8defda3c833dd67107facb0c4
I20260806 03:44:40.414036 12129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.24.60@KRBTEST.COM; defaulting to no policy
I20260806 03:44:40.424381 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.414943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.30.24.62:38975)
0806 03:44:40.414994 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.414997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.415254 (+ 257us) server_negotiation.cc:407] Connection header received
0806 03:44:40.415751 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.415752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.415764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.415828 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.416711 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.417226 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.418830 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.419037 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.419122 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.421710 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.421712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.421712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.421720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.423129 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.423565 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.423566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.423567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.423623 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.423845 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.423846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.423846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.423950 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.424000 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.424242 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.424313 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:44:40.424559 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.414936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38975 (local address 127.0.0.1:55170)
0806 03:44:40.415126 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.415167 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.415641 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.415905 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.415910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.416300 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.416606 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.416612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.417332 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.417334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.418733 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.418737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.419133 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.419135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.419300 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.419879 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.419887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.421612 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.423222 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.423229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.423231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.423497 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.423690 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.423691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.423692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.423790 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.424005 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.424010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.424123 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.424213 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.424333 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.60@KRBTEST.COM" created.
I20260806 03:44:40.425290 12129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:44:40.433053 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.425687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.30.24.61:44033)
0806 03:44:40.425863 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.425867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.425883 (+ 16us) server_negotiation.cc:407] Connection header received
0806 03:44:40.425900 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.425902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.425912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.425996 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.426269 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.426950 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.427620 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.427927 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.428008 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.430210 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.430217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.430217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.430223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.431740 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.432188 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.432191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.432191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.432242 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.432514 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.432515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.432516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.432650 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.432716 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.432905 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.432973 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260806 03:44:40.433218 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.425574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51544)
0806 03:44:40.425623 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.425634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.425678 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.425997 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.425998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.426137 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.426186 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.426188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.427038 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.427039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.427528 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.427530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.428004 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.428005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.428073 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.428528 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.428533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.430110 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.431838 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.431840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.431840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.432095 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.432322 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.432323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.432324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.432404 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.432733 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.432734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.432790 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.432884 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.433069 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":72}
I20260806 03:44:40.433964 12129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:44:40.434353 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.434195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39118)
0806 03:44:40.434254 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.434315 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 03:44:40.434544 12076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
I20260806 03:44:40.435024 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.434920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39124)
0806 03:44:40.434974 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.435002 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 03:44:40.435161 12129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:46097: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab.
Entry for principal kudu/127.30.24.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab.
I20260806 03:44:40.444017 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:46097
--webserver_interface=127.30.24.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab
--principal=kudu/127.30.24.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38975,127.30.24.61:44033,127.30.24.60:46097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:40.482425 12053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } attempt: 3
I20260806 03:44:40.484423 12129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } attempt: 1
I20260806 03:44:40.484869 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.484719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39134)
0806 03:44:40.484779 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.484823 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 03:44:40.485271 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.485143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39142)
0806 03:44:40.485189 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.485215 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 03:44:40.485601 12129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:46097: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:44:40.489995 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.482929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.30.24.61:44033)
0806 03:44:40.483001 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.483003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.483014 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:44:40.483067 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.483068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.483085 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.483154 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.483495 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.484025 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.484864 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.485198 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.485273 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.487432 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.487434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.487434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.487441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.488956 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.489328 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.489329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.489330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.489379 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.489570 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.489571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.489571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.489673 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.489725 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.489879 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.489939 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:44:40.490057 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.482835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51546)
0806 03:44:40.482906 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.482926 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.482997 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.483169 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.483170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.483338 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.483402 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.483404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.484114 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.484116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.484775 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.484778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.485284 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.485286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.485377 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.485906 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.485912 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.487348 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.489037 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.489038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.489039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.489254 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.489444 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.489445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.489446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.489512 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.489736 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.489737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.489787 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.489864 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.489962 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 03:44:40.490545 12053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:44:40.490911 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.490787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39156)
0806 03:44:40.490841 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.490875 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 03:44:40.491297 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.491167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39172)
0806 03:44:40.491214 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.491239 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 03:44:40.491402 12053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:46097: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
I20260806 03:44:40.522535 12053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } attempt: 1
I20260806 03:44:40.523005 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.522868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39180)
0806 03:44:40.522941 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.522972 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 03:44:40.523376 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.523277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39196)
0806 03:44:40.523322 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.523354 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 03:44:40.523473 12053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:46097: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
W20260806 03:44:40.529826 12136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:40.530055 12136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:40.530093 12136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:40.531239 12136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:40.531306 12136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:40.531332 12136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:40.531364 12136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:40.531392 12136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:40.532938 12136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38975,127.30.24.61:44033,127.30.24.60:46097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab
--principal=kudu/127.30.24.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.24.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:40.533215 12136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:40.533455 12136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:40.538219 12142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:40.538276 12141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:40.538219 12144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:40.538481 12136 server_base.cc:1034] running on GCE node
I20260806 03:44:40.538722 12136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:40.539011 12136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:40.540186 12136 hybrid_clock.cc:648] HybridClock initialized: now 1785987880540169 us; error 47 us; skew 500 ppm
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:40.542750 12136 init.cc:377] Logged in from keytab as kudu/127.30.24.60@KRBTEST.COM (short username kudu)
I20260806 03:44:40.543200 12136 webserver.cc:460] Webserver started at http://127.30.24.60:45867/ using document root <none> and password file <none>
I20260806 03:44:40.543442 12136 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:40.543511 12136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:40.543666 12136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:40.544832 12136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data/instance:
uuid: "05eac276df554b35898077b9aa59de3c"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "7d7516feadb9fee91b8be31a61783fc4"
server_key_iv: "1914242afb351ecb195f1e7ec8f0659e"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.545212 12136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal/instance:
uuid: "05eac276df554b35898077b9aa59de3c"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "7d7516feadb9fee91b8be31a61783fc4"
server_key_iv: "1914242afb351ecb195f1e7ec8f0659e"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.546646 12136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 03:44:40.547492 12151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:40.547691 12136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:40.547813 12136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal
uuid: "05eac276df554b35898077b9aa59de3c"
format_stamp: "Formatted at 2026-08-06 03:44:40 on dist-test-slave-10pc"
server_key: "7d7516feadb9fee91b8be31a61783fc4"
server_key_iv: "1914242afb351ecb195f1e7ec8f0659e"
server_key_version: "encryptionkey@0"
I20260806 03:44:40.547900 12136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:40.553779 12129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } attempt: 2
I20260806 03:44:40.554212 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.554064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39198)
0806 03:44:40.554130 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.554175 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 03:44:40.554585 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.554487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39214)
0806 03:44:40.554538 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.554565 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 03:44:40.554703 12129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:46097: Network error: Client connection negotiation failed: client connection to 127.30.24.60:46097: connect: Connection refused (error 111)
I20260806 03:44:40.563596 12136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:40.565742 12136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:40.565919 12136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:40.570384 12136 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.60:46097
I20260806 03:44:40.570423 12204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.60:46097 every 8 connection(s)
I20260806 03:44:40.570803 12136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
I20260806 03:44:40.571673 12205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:40.573237 12205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:40.579777 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12136
I20260806 03:44:40.579918 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785987724420712-30816-0/minicluster-data/master-2/wal/instance
I20260806 03:44:40.580121 30816 external_mini_cluster.cc:1428] Setting key 575f3cd48793d4c331a1c9304b5215ee
I20260806 03:44:40.582679 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.574035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.30.24.62:38975)
0806 03:44:40.574116 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.574118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.574307 (+ 189us) server_negotiation.cc:407] Connection header received
0806 03:44:40.574802 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.574803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.574815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.574867 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.575490 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.575983 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.577023 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.577226 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.577313 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.579298 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.579299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.579300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.579308 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.581114 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.581645 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.581646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.581647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.581749 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.582005 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.582006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.582007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.582162 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.582231 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.582486 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.582577 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:44:40.582882 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.573991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38975 (local address 127.0.0.1:55180)
0806 03:44:40.574190 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.574226 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.574686 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.574937 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.574941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.575182 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.575391 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.575396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.576081 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.576082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.576931 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.576934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.577326 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.577327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.577453 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.577900 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.577908 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.579201 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.581220 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.581228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.581230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.581550 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.581823 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.581823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.581824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.581922 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.582229 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.582234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.582362 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.582443 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.582646 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":73,"thread_start_us":95,"threads_started":1}
I20260806 03:44:40.583431 12205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:44:40.589617 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.580547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.30.24.60:46097)
0806 03:44:40.580708 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.580712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.580764 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:44:40.580837 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.580838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.580849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.580908 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.581646 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.582725 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.583799 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.584153 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.584247 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.586519 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.586526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.586527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.586534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.588301 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.588719 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.588721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.588722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.588775 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.588994 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.588995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.588995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.589115 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.589309 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.589396 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.589471 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260806 03:44:40.590199 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.583692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:44033 (local address 127.0.0.1:51560)
0806 03:44:40.583740 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.583751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.583787 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.583953 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.583954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.584084 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.584130 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.584131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.584959 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.584960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.585449 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.585451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.585825 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.585826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.585895 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.586387 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.586391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.587464 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.589176 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.589177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.589177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.589340 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.589563 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.589564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.589564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.589625 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.589886 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.589887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.589926 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.590011 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.590100 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 03:44:40.590245 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.583784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.30.24.61:44033)
0806 03:44:40.583836 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.583838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.583846 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:44:40.583870 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.583871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.583881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.583949 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.584225 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.584862 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.585543 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.585744 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.585814 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.587554 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.587555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.587556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.587563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.589090 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.589411 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.589412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.589413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.589466 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.589694 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.589695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.589696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.589809 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.589881 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.590057 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.590130 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:44:40.590538 12205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:40.597131 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.590932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39222)
0806 03:44:40.590978 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.590988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.591026 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.591190 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.591190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.591292 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.591321 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.591321 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.592041 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.592042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.592591 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.592593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.592987 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.592988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.593041 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.593396 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.593400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.594489 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.596124 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.596125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.596125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.596347 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.596591 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.596592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.596593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.596658 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.596883 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.596884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.596930 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.597001 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.597074 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 03:44:40.597267 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.590978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.30.24.60:46097)
0806 03:44:40.591014 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.591016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.591025 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:44:40.591102 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.591103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.591114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.591188 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.591390 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.591960 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.592663 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.592919 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.592968 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.594572 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.594573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.594573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.594579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.596043 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.596420 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.596420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.596421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.596524 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.596731 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.596732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.596732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.596824 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.596869 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.597051 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.597124 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:44:40.598193 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.590468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.30.24.62:38975)
0806 03:44:40.590663 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.590664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.590673 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:44:40.590694 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.590695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.590703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.590797 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.591417 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.591995 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.592777 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.592965 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.593030 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.595234 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.595236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.595236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.595244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.596892 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.597399 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.597401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.597401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.597463 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.597689 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.597690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.597690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.597796 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.597991 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.598026 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.598100 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260806 03:44:40.598583 12053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } attempt: 2
I20260806 03:44:40.598892 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: Bootstrap starting.
I20260806 03:44:40.599962 12205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:40.600302 12205 log.cc:824] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:40.601265 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: No bootstrap required, opened a new log
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:44:40.603603 12205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.603760 12205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:40.603839 12205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05eac276df554b35898077b9aa59de3c, State: Initialized, Role: FOLLOWER
I20260806 03:44:40.604017 12205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.604370 12205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:40.605590 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.598801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39230)
0806 03:44:40.598852 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.598860 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.598891 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.599125 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.599126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.599262 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.599291 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.599293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.599860 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.599862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.600379 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.600380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.600892 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.600894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.600951 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.601369 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.601374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.602538 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.604340 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.604341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.604342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.604575 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.604802 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.604803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.604803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.604877 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.605264 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.605266 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.605313 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.605359 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.605453 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260806 03:44:40.605242 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.607115 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:44:40.607194 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.598863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.30.24.61:44033)
0806 03:44:40.598895 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.598897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.598905 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:44:40.598928 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.598929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.598937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.598989 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.599615 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.600252 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.601215 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.601484 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.601529 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.604096 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.604097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.604098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.604106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.605971 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.606361 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.606363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.606363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.606424 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.606654 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.606654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.606655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.606766 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.606961 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.606999 (+ 38us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.607073 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 03:44:40.606387 12222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:44:40.607388 12222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:44:40.608650 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.598988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.30.24.60:46097)
0806 03:44:40.599019 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.599021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.599027 (+ 6us) server_negotiation.cc:407] Connection header received
0806 03:44:40.599043 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.599044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.599051 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.599120 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.599364 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.599782 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.600505 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.600773 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.601110 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.602624 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.602625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.602626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.602633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.604251 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.604664 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.604665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.604666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.604725 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.604968 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.604969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.604969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.605163 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.605732 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.605801 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.605883 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:44:40.609069 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:44:40.606940 12136 master_runner.cc:186] Error getting master registration for 127.30.24.62:38975: OK, instance_id { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" instance_seqno: 1785987880254658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:44:40.609694 12136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:44:40.610441 12053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: Bootstrap starting.
I20260806 03:44:40.611223 12053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:40.611546 12053 log.cc:824] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:40.618986 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.610612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.30.24.60:46097)
0806 03:44:40.610645 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.610646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.610979 (+ 333us) server_negotiation.cc:407] Connection header received
0806 03:44:40.611075 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.611076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.611086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.611114 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.611942 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.612710 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.613673 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.613956 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.613992 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.615560 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.615561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.615562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.615569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.617503 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.618052 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.618053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.618054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.618107 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.618354 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.618355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.618355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.618464 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.618533 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.618834 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.618904 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:44:40.618986 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.611792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.30.24.62:38975)
0806 03:44:40.611872 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.611874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.611883 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:44:40.611902 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.611902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.611911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.611994 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.612558 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.613255 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.614206 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.614478 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.614520 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.615881 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.615882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.615883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.615889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.617493 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.618052 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.618053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.618053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.618108 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.618339 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.618340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.618341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.618463 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.618543 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.618794 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.618895 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:44:40.619328 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.611011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.30.24.61:44033)
0806 03:44:40.611045 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.611047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.611381 (+ 334us) server_negotiation.cc:407] Connection header received
0806 03:44:40.611467 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.611468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.611477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.611554 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.612800 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.613480 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.614464 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.614753 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.614798 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.616249 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.616250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.616250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.616257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.618017 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.618406 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.618407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.618408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.618463 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.618679 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.618680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.618680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.618805 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.618902 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.619167 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.619244 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:44:40.619467 12053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: No bootstrap required, opened a new log
I20260806 03:44:40.621304 12053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.621416 12053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:40.621438 12053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3394ba9548400789d95b9b972b87bf, State: Initialized, Role: FOLLOWER
I20260806 03:44:40.621562 12053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.621833 12232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.621899 12053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:40.621914 12232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: This master's current role is: FOLLOWER
W20260806 03:44:40.623713 12244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:44:40.623798 12244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:44:40.624503 11982 master_runner.cc:186] Error getting master registration for 127.30.24.61:44033: OK, instance_id { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" instance_seqno: 1785987880409413 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:44:40.626879 11982 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:44:40.645875 12129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } attempt: 3
Aug 06 03:44:40 dist-test-slave-10pc krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
I20260806 03:44:40.654664 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.646269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.30.24.60:46097)
0806 03:44:40.646314 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:40.646316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:40.646326 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:44:40.646400 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:40.646401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:40.646411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:40.646472 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:40.646839 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.647580 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.648624 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.648936 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.649003 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.651284 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:40.651285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:40.651286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:40.651293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:40.653422 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.653826 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.653827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.653828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.653893 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:40.654107 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:40.654107 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:40.654108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:40.654247 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:40.654317 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:40.654482 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 03:44:40.654566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:44:40.654657 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:40.646172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:46097 (local address 127.0.0.1:39236)
0806 03:44:40.646265 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:40.646280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:40.646331 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:40.646494 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:40.646495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:40.646702 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:40.646758 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.646760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.647676 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.647677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.648526 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:40.648529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:40.649013 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:40.649015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:40.649076 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:40.649604 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:40.649609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:40.651183 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:40.653507 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.653508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.653509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.653747 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.653966 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:40.653967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:40.653968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:40.654037 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:40.654319 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:40.654320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:40.654371 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 03:44:40.654443 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 03:44:40.654545 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 03:44:40.656793 12129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: Bootstrap starting.
I20260806 03:44:40.657582 12129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:40.657902 12129 log.cc:824] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:40.658804 12129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: No bootstrap required, opened a new log
I20260806 03:44:40.660866 12129 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.661010 12129 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:40.661042 12129 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0bbb92d1504e35b295724665c5f371, State: Initialized, Role: FOLLOWER
I20260806 03:44:40.661219 12129 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.661541 12129 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:40.661500 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.662984 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 03:44:40.664579 12258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:44:40.664633 12258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:44:40.666059 12060 master_runner.cc:263] No leader master found from master be0bbb92d1504e35b295724665c5f371
I20260806 03:44:40.666101 12060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 03:44:40.752245 12004 tablet.cc:2369] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:44:40.816705 12157 tablet.cc:2369] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:44:40.896304 12232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:40.896440 12232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.896821 12232 leader_election.cc:288] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be0bbb92d1504e35b295724665c5f371 (127.30.24.61:44033), 05eac276df554b35898077b9aa59de3c (127.30.24.60:46097)
I20260806 03:44:40.897117 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f3394ba9548400789d95b9b972b87bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05eac276df554b35898077b9aa59de3c" is_pre_election: true
I20260806 03:44:40.897138 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f3394ba9548400789d95b9b972b87bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0bbb92d1504e35b295724665c5f371" is_pre_election: true
I20260806 03:44:40.897305 12178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3394ba9548400789d95b9b972b87bf in term 0.
I20260806 03:44:40.897316 12102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3394ba9548400789d95b9b972b87bf in term 0.
I20260806 03:44:40.897487 12001 leader_election.cc:302] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3394ba9548400789d95b9b972b87bf, be0bbb92d1504e35b295724665c5f371; no voters:
I20260806 03:44:40.897616 12232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:44:40.897681 12232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:40.897704 12232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:40.898460 12232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.898607 12232 leader_election.cc:288] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [CANDIDATE]: Term 1 election: Requested vote from peers be0bbb92d1504e35b295724665c5f371 (127.30.24.61:44033), 05eac276df554b35898077b9aa59de3c (127.30.24.60:46097)
I20260806 03:44:40.898761 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f3394ba9548400789d95b9b972b87bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0bbb92d1504e35b295724665c5f371"
I20260806 03:44:40.898770 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f3394ba9548400789d95b9b972b87bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05eac276df554b35898077b9aa59de3c"
I20260806 03:44:40.898829 12102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:40.898877 12178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:40.899590 12102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3394ba9548400789d95b9b972b87bf in term 1.
I20260806 03:44:40.899600 12178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3394ba9548400789d95b9b972b87bf in term 1.
I20260806 03:44:40.899763 12001 leader_election.cc:302] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3394ba9548400789d95b9b972b87bf, be0bbb92d1504e35b295724665c5f371; no voters:
I20260806 03:44:40.899870 12232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:40.900017 12232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [term 1 LEADER]: Becoming Leader. State: Replica: 8f3394ba9548400789d95b9b972b87bf, State: Running, Role: LEADER
I20260806 03:44:40.900120 12232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } }
I20260806 03:44:40.900534 12261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3394ba9548400789d95b9b972b87bf. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.900630 12261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:40.900825 12264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:40.901007 12264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:44:40.902408 12178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [term 1 FOLLOWER]: Refusing update from remote peer 8f3394ba9548400789d95b9b972b87bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:40.902422 12102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [term 1 FOLLOWER]: Refusing update from remote peer 8f3394ba9548400789d95b9b972b87bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:40.902720 12261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:40.902822 12232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:40.904060 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3394ba9548400789d95b9b972b87bf. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.904102 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3394ba9548400789d95b9b972b87bf. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.904170 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:44:40.904181 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:44:40.904986 12261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.905100 12261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:40.905166 12261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.905210 12261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:40.905225 12267 mvcc.cc:205] Tried to move back new op lower bound from 7315406360173789184 to 7315406360167583744. Current Snapshot: MvccSnapshot[applied={T|T < 7315406360173789184}]
I20260806 03:44:40.905347 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.905428 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:44:40.905506 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f3394ba9548400789d95b9b972b87bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3394ba9548400789d95b9b972b87bf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38975 } } peers { permanent_uuid: "be0bbb92d1504e35b295724665c5f371" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 44033 } } peers { permanent_uuid: "05eac276df554b35898077b9aa59de3c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 46097 } } }
I20260806 03:44:40.905532 12264 catalog_manager.cc:1347] Generated new cluster ID: 352a4b908e644ee38e1e6ecbe52dd0e8
I20260806 03:44:40.905578 12264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:44:40.905573 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:44:40.905704 12266 mvcc.cc:205] Tried to move back new op lower bound from 7315406360173789184 to 7315406360167583744. Current Snapshot: MvccSnapshot[applied={T|T < 7315406360173789184}]
I20260806 03:44:40.909505 12264 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:44:40.910184 12264 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:44:40.915658 12264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f3394ba9548400789d95b9b972b87bf: Generated new TSK 0
I20260806 03:44:40.915889 12264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:44:41.607904 12222 catalog_manager.cc:1259] Loaded cluster ID: 352a4b908e644ee38e1e6ecbe52dd0e8
I20260806 03:44:41.607978 12222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: loading cluster ID for follower catalog manager: success
I20260806 03:44:41.608774 12222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: acquiring CA information for follower catalog manager: success
I20260806 03:44:41.609036 12222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05eac276df554b35898077b9aa59de3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:44:41.665138 12258 catalog_manager.cc:1259] Loaded cluster ID: 352a4b908e644ee38e1e6ecbe52dd0e8
I20260806 03:44:41.665238 12258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: loading cluster ID for follower catalog manager: success
I20260806 03:44:41.666138 12258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: acquiring CA information for follower catalog manager: success
I20260806 03:44:41.666471 12258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be0bbb92d1504e35b295724665c5f371: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:44:41.980187 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 11982
I20260806 03:44:41.987519 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12060
I20260806 03:44:41.993848 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12136
2026-08-06T03:44:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3069 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:44:42 dist-test-slave-10pc krb5kdc[12279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:44:42 dist-test-slave-10pc krb5kdc[12279](info): set up 2 sockets
Aug 06 03:44:42 dist-test-slave-10pc krb5kdc[12279](info): commencing operation
krb5kdc: starting...
W20260806 03:44:43.045433 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:44:43 dist-test-slave-10pc krb5kdc[12279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:44:43.170847 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38347
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:43.272270 12292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:43.272533 12292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:43.272586 12292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:43.273730 12292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:43.273772 12292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:43.273787 12292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:43.273839 12292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:43.273856 12292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:43.275326 12292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:43.275578 12292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:43.275811 12292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:43.280714 12298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:43.280709 12300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:43.280772 12297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:43.281020 12292 server_base.cc:1034] running on GCE node
I20260806 03:44:43.281286 12292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:43.281572 12292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:43.282738 12292 hybrid_clock.cc:648] HybridClock initialized: now 1785987883282726 us; error 47 us; skew 500 ppm
Aug 06 03:44:43 dist-test-slave-10pc krb5kdc[12279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:43.285501 12292 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:43.285946 12292 webserver.cc:460] Webserver started at http://127.30.24.62:35411/ using document root <none> and password file <none>
I20260806 03:44:43.286204 12292 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:43.286278 12292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:43.286432 12292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:43.287593 12292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "f7844a47dd4d49a5bbc5e077e9abd964"
format_stamp: "Formatted at 2026-08-06 03:44:43 on dist-test-slave-10pc"
server_key: "c25d7c4a008380d9f826083b63903e8f"
server_key_iv: "1502c402387be7dfd1f17bcf721323e4"
server_key_version: "encryptionkey@0"
I20260806 03:44:43.287999 12292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "f7844a47dd4d49a5bbc5e077e9abd964"
format_stamp: "Formatted at 2026-08-06 03:44:43 on dist-test-slave-10pc"
server_key: "c25d7c4a008380d9f826083b63903e8f"
server_key_iv: "1502c402387be7dfd1f17bcf721323e4"
server_key_version: "encryptionkey@0"
I20260806 03:44:43.289405 12292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:44:43.290138 12307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:43.290334 12292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:44:43.290401 12292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "f7844a47dd4d49a5bbc5e077e9abd964"
format_stamp: "Formatted at 2026-08-06 03:44:43 on dist-test-slave-10pc"
server_key: "c25d7c4a008380d9f826083b63903e8f"
server_key_iv: "1502c402387be7dfd1f17bcf721323e4"
server_key_version: "encryptionkey@0"
I20260806 03:44:43.290498 12292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:43.305625 12292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:43.307694 12292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:43.307840 12292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:43.313024 12292 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:38347
I20260806 03:44:43.313062 12362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:38347 every 8 connection(s)
I20260806 03:44:43.313510 12292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:43.314683 12363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:43.316658 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12292
I20260806 03:44:43.316771 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:44:43.316960 30816 external_mini_cluster.cc:1428] Setting key e87756602aa9aaf3d20c221149ba14a5
I20260806 03:44:43.318838 12363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964: Bootstrap starting.
I20260806 03:44:43.319612 12363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:43.319962 12363 log.cc:824] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:43.320860 12363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964: No bootstrap required, opened a new log
I20260806 03:44:43.323637 12363 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38347 } }
I20260806 03:44:43.323835 12363 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:43.323881 12363 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7844a47dd4d49a5bbc5e077e9abd964, State: Initialized, Role: FOLLOWER
I20260806 03:44:43.324035 12363 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38347 } }
I20260806 03:44:43.324132 12363 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:44:43.324186 12363 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:44:43.324239 12363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:43.325062 12363 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38347 } }
I20260806 03:44:43.325194 12363 leader_election.cc:302] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7844a47dd4d49a5bbc5e077e9abd964; no voters:
I20260806 03:44:43.325379 12363 leader_election.cc:288] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:44:43.325433 12368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:43.325584 12368 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [term 1 LEADER]: Becoming Leader. State: Replica: f7844a47dd4d49a5bbc5e077e9abd964, State: Running, Role: LEADER
I20260806 03:44:43.325652 12363 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:43.325687 12368 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38347 } }
Aug 06 03:44:43 dist-test-slave-10pc krb5kdc[12279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:43.325992 12368 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7844a47dd4d49a5bbc5e077e9abd964. Latest consensus state: current_term: 1 leader_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38347 } } }
I20260806 03:44:43.326076 12368 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:43.326001 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7844a47dd4d49a5bbc5e077e9abd964" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38347 } } }
I20260806 03:44:43.326264 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964 [sys.catalog]: This master's current role is: LEADER
W20260806 03:44:43.327454 12383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:44:43.327512 12383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:44:43.327757 12374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:43.327934 12374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:44:43.329872 12374 catalog_manager.cc:1347] Generated new cluster ID: 0641811295b44ab797a676830725df18
I20260806 03:44:43.329921 12374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:44:43.330221 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:43.317694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.30.24.62:38347)
0806 03:44:43.317912 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:43.317924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:43.317960 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:44:43.318673 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:43.318684 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:43.318756 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:43.318919 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:43.319837 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:43.321495 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:43.322578 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:43.323257 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:43.323374 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:43.326478 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:43.326482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:43.326482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:43.326493 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:43.328596 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:43.328981 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:43.328983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:43.328983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:43.329066 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:43.329329 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:43.329330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:43.329331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:43.329614 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:43.329799 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:43.329900 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 03:44:43.330055 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260806 03:44:43.333408 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:44:43.343518 12374 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:44:43.344208 12374 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:44:43.351575 12374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7844a47dd4d49a5bbc5e077e9abd964: Generated new TSK 0
I20260806 03:44:43.351750 12374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:44:43.387585 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12292
2026-08-06T03:44:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1392 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:44:43 dist-test-slave-10pc krb5kdc[12390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:44:43 dist-test-slave-10pc krb5kdc[12390](info): set up 2 sockets
Aug 06 03:44:43 dist-test-slave-10pc krb5kdc[12390](info): commencing operation
krb5kdc: starting...
W20260806 03:44:44.425503 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:44:44 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:44:44.552575 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34817
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:34817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:44.636853 12404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:44.637108 12404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:44.637149 12404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:44.638291 12404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:44.638361 12404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:44.638386 12404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:44.638406 12404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:44.638437 12404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:44.640564 12404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:34817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:44.640851 12404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:44.641085 12404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:44.646209 12412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:44.646289 12409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:44.646219 12410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:44.646373 12404 server_base.cc:1034] running on GCE node
I20260806 03:44:44.646710 12404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:44.646975 12404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:44.648138 12404 hybrid_clock.cc:648] HybridClock initialized: now 1785987884648116 us; error 40 us; skew 500 ppm
Aug 06 03:44:44 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:44.650770 12404 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:44.651325 12404 webserver.cc:460] Webserver started at http://127.30.24.62:38013/ using document root <none> and password file <none>
I20260806 03:44:44.651597 12404 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:44.651649 12404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:44.651816 12404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:44.653116 12404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "dd0093b7a5df43e2901e592b4f575852"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "6185ce5d5210452ff3a2afd880e13848"
server_key_iv: "80bb5116e34777a07ea4841f644b2e45"
server_key_version: "encryptionkey@0"
I20260806 03:44:44.653535 12404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "dd0093b7a5df43e2901e592b4f575852"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "6185ce5d5210452ff3a2afd880e13848"
server_key_iv: "80bb5116e34777a07ea4841f644b2e45"
server_key_version: "encryptionkey@0"
I20260806 03:44:44.655055 12404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:44:44.655920 12419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:44.656224 12404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:44.656337 12404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "dd0093b7a5df43e2901e592b4f575852"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "6185ce5d5210452ff3a2afd880e13848"
server_key_iv: "80bb5116e34777a07ea4841f644b2e45"
server_key_version: "encryptionkey@0"
I20260806 03:44:44.656419 12404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:44.683917 12404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:44.686051 12404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:44.686235 12404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:44.690608 12404 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:34817
I20260806 03:44:44.690693 12472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:34817 every 8 connection(s)
I20260806 03:44:44.691063 12404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:44.692131 12473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:44.694751 12473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap starting.
I20260806 03:44:44.695348 12473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:44.695606 12473 log.cc:824] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:44.696389 12473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: No bootstrap required, opened a new log
I20260806 03:44:44.698177 12473 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:44.698334 12473 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:44.698359 12473 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0093b7a5df43e2901e592b4f575852, State: Initialized, Role: FOLLOWER
I20260806 03:44:44.698513 12473 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:44.698604 12473 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:44:44.698637 12473 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:44:44.698675 12473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:44.698863 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12404
I20260806 03:44:44.698992 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:44:44.699211 30816 external_mini_cluster.cc:1428] Setting key 4bafe477783a6f05d98885f2aacb1262
I20260806 03:44:44.699946 12473 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:44.700120 12473 leader_election.cc:302] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0093b7a5df43e2901e592b4f575852; no voters:
I20260806 03:44:44.700316 12473 leader_election.cc:288] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:44:44.700336 12479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:44.700543 12479 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 LEADER]: Becoming Leader. State: Replica: dd0093b7a5df43e2901e592b4f575852, State: Running, Role: LEADER
I20260806 03:44:44.700621 12473 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:44.700706 12479 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:44.701141 12480 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0093b7a5df43e2901e592b4f575852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } } }
I20260806 03:44:44.701234 12480 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:44.701141 12481 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0093b7a5df43e2901e592b4f575852. Latest consensus state: current_term: 1 leader_uuid: "dd0093b7a5df43e2901e592b4f575852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } } }
I20260806 03:44:44.701515 12481 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:44.702046 12486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:44.702642 12486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:44:44.704429 12486 catalog_manager.cc:1347] Generated new cluster ID: a9e731f91b894e53a47b505ed561b48a
I20260806 03:44:44.704550 12486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:44:44 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:44.710938 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:44.699779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.30.24.62:34817)
0806 03:44:44.700036 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:44.700048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:44.700101 (+ 53us) server_negotiation.cc:407] Connection header received
0806 03:44:44.700237 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:44.700246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:44.700315 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:44.700562 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:44.701391 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:44.702820 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:44.703867 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:44.704418 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:44.704581 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:44.707013 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:44.707017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:44.707017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:44.707029 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:44.709352 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:44.709741 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:44.709745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:44.709745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:44.709854 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:44.710085 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:44.710086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:44.710086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:44.710328 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:44.710519 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:44.710589 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 03:44:44.710733 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":106,"threads_started":1}
I20260806 03:44:44.718683 12486 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:44:44.719381 12486 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:44:44.723083 12486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Generated new TSK 0
I20260806 03:44:44.723265 12486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:44:44.756381 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34817
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:44.883157 12500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:44.883407 12500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:44.883448 12500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:44.884721 12500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:44.884781 12500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:44.884827 12500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:44:44.886372 12500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:44.886656 12500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:44.886895 12500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:44.893095 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:44.893203 12505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:44.893108 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:44.893494 12500 server_base.cc:1034] running on GCE node
I20260806 03:44:44.893718 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:44.894022 12500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:44.895206 12500 hybrid_clock.cc:648] HybridClock initialized: now 1785987884895192 us; error 27 us; skew 500 ppm
Aug 06 03:44:44 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:44.898260 12500 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:44:44.898828 12500 webserver.cc:460] Webserver started at http://127.30.24.1:46253/ using document root <none> and password file <none>
I20260806 03:44:44.899108 12500 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:44.899168 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:44.899353 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:44.900622 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "a69d5f7783254b9c9ab30dd72336db9e"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "7c0ea289514625a817a3d337b6fe7f1c"
server_key_iv: "fffc8b3b12889b5e7c4ea4984fdd3779"
server_key_version: "encryptionkey@0"
I20260806 03:44:44.901091 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "a69d5f7783254b9c9ab30dd72336db9e"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "7c0ea289514625a817a3d337b6fe7f1c"
server_key_iv: "fffc8b3b12889b5e7c4ea4984fdd3779"
server_key_version: "encryptionkey@0"
I20260806 03:44:44.902592 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:44:44.903384 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:44.903611 12500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:44.903692 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "a69d5f7783254b9c9ab30dd72336db9e"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "7c0ea289514625a817a3d337b6fe7f1c"
server_key_iv: "fffc8b3b12889b5e7c4ea4984fdd3779"
server_key_version: "encryptionkey@0"
I20260806 03:44:44.903750 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:44.945776 12500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:44.949016 12500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:44.949160 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:44.949422 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:44.949774 12500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:44.949846 12500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:44.949890 12500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:44.949919 12500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:44.956080 12500 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:42217
I20260806 03:44:44.956116 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:42217 every 8 connection(s)
I20260806 03:44:44.956589 12500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:44:44 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:44.964421 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12500
I20260806 03:44:44.964555 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:44:44.964776 30816 external_mini_cluster.cc:1428] Setting key 562488a37b6c0f823d89f91d9cd45536
I20260806 03:44:44.967116 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:44.957476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:59201 (local address 127.30.24.62:34817)
0806 03:44:44.957607 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:44.957610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:44.957781 (+ 171us) server_negotiation.cc:407] Connection header received
0806 03:44:44.958430 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:44.958432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:44.958447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:44.958492 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:44.959598 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:44.960168 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:44.961366 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:44.961616 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:44.961714 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:44.963974 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:44.963976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:44.963976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:44.963983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:44.965644 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:44.966145 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:44.966146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:44.966146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:44.966198 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:44.966487 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:44.966487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:44.966488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:44.966589 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:44.966638 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:44.966983 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 03:44:44.967051 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 03:44:44.967446 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:44.957430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:59201)
0806 03:44:44.957637 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:44.957690 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:44.958298 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:44.958587 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:44.958591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:44.959243 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:44.959464 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:44.959469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:44.960279 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:44.960280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:44.961247 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:44.961249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:44.961728 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:44.961729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:44.961868 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:44.962443 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:44.962451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:44.963881 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:44.965732 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:44.965741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:44.965744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:44.966064 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:44.966263 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:44.966264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:44.966265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:44.966423 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:44.966653 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:44.966660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:44.966842 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 03:44:44.966963 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 03:44:44.967156 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260806 03:44:44.967960 12634 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34817
I20260806 03:44:44.968086 12634 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:44.968305 12634 heartbeater.cc:511] Master 127.30.24.62:34817 requested a full tablet report, sending...
I20260806 03:44:44.968888 12437 ts_manager.cc:194] Registered new tserver with Master: a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217)
I20260806 03:44:44.969520 12437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:59201
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:44:44.993578 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34817
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:45.080145 12639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:45.080374 12639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:45.080399 12639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:45.081573 12639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:45.081616 12639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:45.081643 12639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:44:45.083005 12639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:45.083204 12639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:45.083381 12639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:45.088370 12645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.088378 12644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.088378 12647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:45.089262 12639 server_base.cc:1034] running on GCE node
I20260806 03:44:45.089483 12639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:45.089738 12639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:45.090898 12639 hybrid_clock.cc:648] HybridClock initialized: now 1785987885090882 us; error 26 us; skew 500 ppm
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:45.093582 12639 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:44:45.094075 12639 webserver.cc:460] Webserver started at http://127.30.24.2:45897/ using document root <none> and password file <none>
I20260806 03:44:45.094342 12639 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:45.094409 12639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:45.094550 12639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:45.095633 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "c45122be1aec43ecaa5491ce834263d7"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "95d73a458f820352b1c2dceaf4cd309d"
server_key_iv: "3697bc413f5701e3835851e7188089a0"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.096006 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "c45122be1aec43ecaa5491ce834263d7"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "95d73a458f820352b1c2dceaf4cd309d"
server_key_iv: "3697bc413f5701e3835851e7188089a0"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.097472 12639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:44:45.098189 12654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.098361 12639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:44:45.098464 12639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "c45122be1aec43ecaa5491ce834263d7"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "95d73a458f820352b1c2dceaf4cd309d"
server_key_iv: "3697bc413f5701e3835851e7188089a0"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.098536 12639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:45.121480 12639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:45.123579 12639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:45.123749 12639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:45.123996 12639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:45.124315 12639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:45.124367 12639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.124408 12639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:45.124439 12639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.130497 12639 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:45081
I20260806 03:44:45.130546 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:45081 every 8 connection(s)
I20260806 03:44:45.130864 12639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:45.139699 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12639
I20260806 03:44:45.139827 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:44:45.140054 30816 external_mini_cluster.cc:1428] Setting key bffd106fa5a829789be8f6c0dee71ab7
I20260806 03:44:45.140204 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.131590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:49077 (local address 127.30.24.62:34817)
0806 03:44:45.131662 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.131664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.131909 (+ 245us) server_negotiation.cc:407] Connection header received
0806 03:44:45.132411 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.132412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.132422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.132526 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:45.133466 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.133960 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.134935 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.135120 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.135180 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.137490 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:45.137492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:45.137492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:45.137498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:45.138789 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.139190 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.139191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.139191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.139241 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.139459 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.139459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.139459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.139568 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:45.139618 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.139995 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.140073 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 03:44:45.140427 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.131572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:49077)
0806 03:44:45.131781 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.131825 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.132328 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.132595 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.132598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.133167 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:45.133384 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.133389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.134038 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.134039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.134863 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.134865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.135212 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.135213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.135319 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.135868 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:45.135877 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:45.137384 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:45.138870 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.138877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.138879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.139113 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.139299 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.139299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.139300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.139405 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.139644 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:45.139652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:45.139845 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.139980 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.140168 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260806 03:44:45.140933 12772 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34817
I20260806 03:44:45.141036 12772 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:45.141228 12772 heartbeater.cc:511] Master 127.30.24.62:34817 requested a full tablet report, sending...
I20260806 03:44:45.141635 12437 ts_manager.cc:194] Registered new tserver with Master: c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081)
I20260806 03:44:45.142140 12437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:49077
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:44:45.167207 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34817
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:45.252787 12777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:45.253001 12777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:45.253026 12777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:45.254137 12777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:45.254171 12777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:45.254197 12777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:44:45.255590 12777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:45.255793 12777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:45.255981 12777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:45.261199 12783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.261206 12782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.261230 12785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:45.261490 12777 server_base.cc:1034] running on GCE node
I20260806 03:44:45.261714 12777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:45.261955 12777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:45.263113 12777 hybrid_clock.cc:648] HybridClock initialized: now 1785987885263105 us; error 35 us; skew 500 ppm
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:45.270640 12777 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:44:45.271131 12777 webserver.cc:460] Webserver started at http://127.30.24.3:43575/ using document root <none> and password file <none>
I20260806 03:44:45.271333 12777 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:45.271375 12777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:45.271471 12777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:44:45.272655 12777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "7103a0b492924f59932fb88796259587"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "513078dd71b807ada4f8bc698497c941"
server_key_iv: "dc0305a8292cb89bd5113ac686b9558a"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.272981 12777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "7103a0b492924f59932fb88796259587"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "513078dd71b807ada4f8bc698497c941"
server_key_iv: "dc0305a8292cb89bd5113ac686b9558a"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.274411 12777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:44:45.275189 12792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.275383 12777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:45.275480 12777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "7103a0b492924f59932fb88796259587"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "513078dd71b807ada4f8bc698497c941"
server_key_iv: "dc0305a8292cb89bd5113ac686b9558a"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.275542 12777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:45.288443 12777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:45.291030 12777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:45.291157 12777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:45.291358 12777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:45.291791 12777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:45.291833 12777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.291873 12777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:45.291900 12777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.297827 12777 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37579
I20260806 03:44:45.297890 12909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37579 every 8 connection(s)
I20260806 03:44:45.298224 12777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:44:45.303001 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12777
I20260806 03:44:45.303115 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:44:45.303335 30816 external_mini_cluster.cc:1428] Setting key 7b1a52f75b922d878ed29643aebde36b
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:45.308269 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.298977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:39661 (local address 127.30.24.62:34817)
0806 03:44:45.299045 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.299047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.299245 (+ 198us) server_negotiation.cc:407] Connection header received
0806 03:44:45.299749 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.299750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.299761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.299825 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:45.300904 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.301621 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.302646 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.302832 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.302889 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.305394 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:45.305396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:45.305396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:45.305404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:45.307028 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.307460 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.307461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.307462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.307519 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.307742 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.307742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.307743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.307853 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:45.307905 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.308138 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.308204 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 03:44:45.308501 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.298933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:39661)
0806 03:44:45.299107 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.299151 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.299663 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.299887 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.299890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.300523 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:45.300786 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.300792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.301710 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.301711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.302558 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.302560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.302914 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.302916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.303068 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.303707 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:45.303717 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:45.305298 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:45.307109 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.307117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.307120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.307364 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.307581 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.307581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.307582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.307683 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.307906 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:45.307911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:45.308031 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.308113 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.308235 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260806 03:44:45.309098 12910 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34817
I20260806 03:44:45.309212 12910 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:45.309406 12910 heartbeater.cc:511] Master 127.30.24.62:34817 requested a full tablet report, sending...
I20260806 03:44:45.309813 12437 ts_manager.cc:194] Registered new tserver with Master: 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
I20260806 03:44:45.310245 12437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:39661
I20260806 03:44:45.315119 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:44:45.322537 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.316498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.30.24.62:34817)
0806 03:44:45.316556 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.316559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.316687 (+ 128us) server_negotiation.cc:407] Connection header received
0806 03:44:45.316735 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.316735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.316746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.316796 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:45.317339 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.317821 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.318522 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.318711 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.318770 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.320085 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:45.320087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:45.320087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:45.320094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:45.321445 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.321804 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.321805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.321805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.321861 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.322031 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.322031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.322032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.322207 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:45.322284 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.322381 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.322443 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:44:45.324286 12437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:44:45.325997 12437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:44:45.333415 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.328726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:45081 (local address 127.0.0.1:34602)
0806 03:44:45.329138 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.329185 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.329268 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.329674 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.329677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.329690 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:45.329781 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.329784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.331134 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.331135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.332250 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.332252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.333045 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.333046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.333143 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.333157 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.333256 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.333317 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":6,"thread_start_us":82,"threads_started":1}
I20260806 03:44:45.333423 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.328946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.30.24.2:45081)
0806 03:44:45.329178 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.329184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.329360 (+ 176us) server_negotiation.cc:407] Connection header received
0806 03:44:45.329413 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.329415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.329523 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.329590 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:45.329861 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.331026 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.332335 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.332935 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.333039 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.333108 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.333275 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.333317 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
I20260806 03:44:45.334492 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.328736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37579 (local address 127.0.0.1:44050)
0806 03:44:45.329152 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.329211 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.329268 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.329674 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.329677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.329679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:45.329781 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.329784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.332150 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.332151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.333295 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.333298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.334224 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.334225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.334304 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.334309 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.334371 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.334412 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":31,"thread_start_us":104,"threads_started":1}
I20260806 03:44:45.334537 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.328854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.30.24.3:37579)
0806 03:44:45.329422 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.329426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.329436 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:44:45.329474 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.329476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.329491 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.329569 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:45.331170 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.332053 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.333394 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.334135 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.334221 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.334276 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.334397 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.334439 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":110,"threads_started":1}
I20260806 03:44:45.334424 12704 tablet_service.cc:1467] Processing CreateTablet for tablet dfba77cca6ea4155bb512d7edde43238 (DEFAULT_TABLE table=test-table [id=b2f61c79383d46e395c49128aed64419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:45.334872 12704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfba77cca6ea4155bb512d7edde43238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:45.335083 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.328969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:42217 (local address 127.0.0.1:54490)
0806 03:44:45.329239 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.329251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.329311 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.329734 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.329735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.329737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:45.329815 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.329817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.332350 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.332351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.333439 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.333441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.334844 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.334845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.334901 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.334905 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.334971 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.335016 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260806 03:44:45.335309 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.329114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.30.24.1:42217)
0806 03:44:45.329529 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.329534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.329544 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:44:45.329597 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.329599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.329615 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.329727 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:45.329905 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.331034 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.333752 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.334487 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.334556 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.334802 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.335186 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.335229 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":88,"threads_started":1}
I20260806 03:44:45.335390 12842 tablet_service.cc:1467] Processing CreateTablet for tablet dfba77cca6ea4155bb512d7edde43238 (DEFAULT_TABLE table=test-table [id=b2f61c79383d46e395c49128aed64419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:45.335728 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfba77cca6ea4155bb512d7edde43238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:45.336006 12565 tablet_service.cc:1467] Processing CreateTablet for tablet dfba77cca6ea4155bb512d7edde43238 (DEFAULT_TABLE table=test-table [id=b2f61c79383d46e395c49128aed64419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:45.336297 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfba77cca6ea4155bb512d7edde43238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:45.337919 12928 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Bootstrap starting.
I20260806 03:44:45.338558 12928 tablet_bootstrap.cc:654] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:45.338996 12928 log.cc:824] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:45.339578 12930 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Bootstrap starting.
I20260806 03:44:45.339730 12932 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap starting.
I20260806 03:44:45.340307 12930 tablet_bootstrap.cc:654] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:45.340572 12932 tablet_bootstrap.cc:654] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:45.340924 12930 log.cc:824] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:45.341162 12932 log.cc:824] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:45.341195 12928 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: No bootstrap required, opened a new log
I20260806 03:44:45.341257 12928 ts_tablet_manager.cc:1397] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 03:44:45.341686 12930 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: No bootstrap required, opened a new log
I20260806 03:44:45.341769 12930 ts_tablet_manager.cc:1397] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:44:45.341920 12932 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: No bootstrap required, opened a new log
I20260806 03:44:45.342000 12932 ts_tablet_manager.cc:1397] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:44:45.343205 12930 raft_consensus.cc:348] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.343365 12930 raft_consensus.cc:374] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:45.343429 12930 raft_consensus.cc:729] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7103a0b492924f59932fb88796259587, State: Initialized, Role: FOLLOWER
I20260806 03:44:45.343343 12928 raft_consensus.cc:348] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.343550 12928 raft_consensus.cc:374] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:45.343578 12930 consensus_queue.cc:260] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.343623 12928 raft_consensus.cc:729] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45122be1aec43ecaa5491ce834263d7, State: Initialized, Role: FOLLOWER
I20260806 03:44:45.343798 12928 consensus_queue.cc:260] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.343889 12930 ts_tablet_manager.cc:1428] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:44:45.343910 12910 heartbeater.cc:503] Master 127.30.24.62:34817 was elected leader, sending a full tablet report...
I20260806 03:44:45.344007 12932 raft_consensus.cc:348] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.344136 12928 ts_tablet_manager.cc:1428] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:44:45.344158 12932 raft_consensus.cc:374] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:45.344192 12932 raft_consensus.cc:729] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d5f7783254b9c9ab30dd72336db9e, State: Initialized, Role: FOLLOWER
I20260806 03:44:45.344246 12772 heartbeater.cc:503] Master 127.30.24.62:34817 was elected leader, sending a full tablet report...
I20260806 03:44:45.344398 12932 consensus_queue.cc:260] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.344722 12932 ts_tablet_manager.cc:1428] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 03:44:45.344878 12634 heartbeater.cc:503] Master 127.30.24.62:34817 was elected leader, sending a full tablet report...
W20260806 03:44:45.381412 12773 tablet.cc:2369] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:44:45.402207 12935 raft_consensus.cc:482] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:45.402397 12935 raft_consensus.cc:504] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.402871 12935 leader_election.cc:288] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217)
I20260806 03:44:45.407060 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.403045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:42217 (local address 127.30.24.3:49013)
0806 03:44:45.403112 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.403128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.403189 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.403487 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.403488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.403489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:45.403569 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.403571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.404610 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.404611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.405947 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.405951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.406746 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.406747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.406833 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.406838 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.406896 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.406950 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":44}
I20260806 03:44:45.407084 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.403226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:49013 (local address 127.30.24.1:42217)
0806 03:44:45.403336 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.403339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.403348 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:44:45.403381 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.403382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.403395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.403464 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:45.403641 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.404521 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.406068 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.406661 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.406743 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.406755 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.406966 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.407009 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 03:44:45.407084 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.403189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:44189 (local address 127.30.24.2:45081)
0806 03:44:45.403278 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.403281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.403364 (+ 83us) server_negotiation.cc:407] Connection header received
0806 03:44:45.403417 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.403418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.403432 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.403497 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:45.403638 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.404521 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.406068 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.406657 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.406743 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.406755 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.406962 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.407009 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 03:44:45.407070 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.403045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:45081 (local address 127.30.24.3:44189)
0806 03:44:45.403297 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.403307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.403369 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.403510 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.403511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.403512 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:45.403569 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.403571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.404610 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.404611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.405947 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.405951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.406746 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.406747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.406860 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.406864 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.406944 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.406977 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":89,"thread_start_us":131,"threads_started":1}
I20260806 03:44:45.407431 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfba77cca6ea4155bb512d7edde43238" candidate_uuid: "7103a0b492924f59932fb88796259587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d5f7783254b9c9ab30dd72336db9e" is_pre_election: true
I20260806 03:44:45.407445 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfba77cca6ea4155bb512d7edde43238" candidate_uuid: "7103a0b492924f59932fb88796259587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45122be1aec43ecaa5491ce834263d7" is_pre_election: true
I20260806 03:44:45.407620 12587 raft_consensus.cc:2454] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7103a0b492924f59932fb88796259587 in term 0.
I20260806 03:44:45.407629 12725 raft_consensus.cc:2454] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7103a0b492924f59932fb88796259587 in term 0.
I20260806 03:44:45.407862 12796 leader_election.cc:302] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7103a0b492924f59932fb88796259587, c45122be1aec43ecaa5491ce834263d7; no voters:
I20260806 03:44:45.408030 12935 raft_consensus.cc:2781] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:44:45.408118 12935 raft_consensus.cc:482] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:45.408154 12935 raft_consensus.cc:3037] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:45.408982 12935 raft_consensus.cc:504] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.409142 12935 leader_election.cc:288] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [CANDIDATE]: Term 1 election: Requested vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217)
I20260806 03:44:45.409315 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfba77cca6ea4155bb512d7edde43238" candidate_uuid: "7103a0b492924f59932fb88796259587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d5f7783254b9c9ab30dd72336db9e"
I20260806 03:44:45.409345 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfba77cca6ea4155bb512d7edde43238" candidate_uuid: "7103a0b492924f59932fb88796259587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45122be1aec43ecaa5491ce834263d7"
I20260806 03:44:45.409399 12587 raft_consensus.cc:3037] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:45.409423 12725 raft_consensus.cc:3037] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:45.410136 12725 raft_consensus.cc:2454] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7103a0b492924f59932fb88796259587 in term 1.
I20260806 03:44:45.410143 12587 raft_consensus.cc:2454] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7103a0b492924f59932fb88796259587 in term 1.
I20260806 03:44:45.410326 12794 leader_election.cc:302] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7103a0b492924f59932fb88796259587, a69d5f7783254b9c9ab30dd72336db9e; no voters:
I20260806 03:44:45.410436 12935 raft_consensus.cc:2781] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:45.410627 12935 raft_consensus.cc:686] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 1 LEADER]: Becoming Leader. State: Replica: 7103a0b492924f59932fb88796259587, State: Running, Role: LEADER
I20260806 03:44:45.410732 12935 consensus_queue.cc:237] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.411660 12435 catalog_manager.cc:5495] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7103a0b492924f59932fb88796259587 (127.30.24.3). New cstate: current_term: 1 leader_uuid: "7103a0b492924f59932fb88796259587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } health_report { overall_health: UNKNOWN } } }
I20260806 03:44:45.425364 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.422983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.30.24.3:37579)
0806 03:44:45.423046 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.423048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.423056 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:44:45.423088 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.423089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.423109 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.423187 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:44:45.423308 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.423770 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.424618 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.424862 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.424916 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.425066 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:44:45.425166 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:44:45.425227 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.425268 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.425287 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:44:45.427163 12725 raft_consensus.cc:1261] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Refusing update from remote peer 7103a0b492924f59932fb88796259587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:45.427172 12587 raft_consensus.cc:1261] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Refusing update from remote peer 7103a0b492924f59932fb88796259587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:45.427529 12935 consensus_queue.cc:1035] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:45.427685 12939 consensus_queue.cc:1035] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:45.429991 12942 mvcc.cc:205] Tried to move back new op lower bound from 7315406378707128320 to 7315406378643197952. Current Snapshot: MvccSnapshot[applied={T|T < 7315406378707128320}]
I20260806 03:44:45.431480 12944 mvcc.cc:205] Tried to move back new op lower bound from 7315406378707128320 to 7315406378643197952. Current Snapshot: MvccSnapshot[applied={T|T < 7315406378707128320 or (T in {7315406378707128320})}]
I20260806 03:44:45.432665 12435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:44:45.432773 12435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:44:45.433527 12435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Sending DeleteTablet for 3 replicas of tablet dfba77cca6ea4155bb512d7edde43238
I20260806 03:44:45.433877 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet dfba77cca6ea4155bb512d7edde43238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:45 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:34602
I20260806 03:44:45.433941 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12500
I20260806 03:44:45.433929 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet dfba77cca6ea4155bb512d7edde43238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:45 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:54490
I20260806 03:44:45.434911 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet dfba77cca6ea4155bb512d7edde43238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:45 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:44050
I20260806 03:44:45.435165 12954 tablet_replica.cc:330] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: stopping tablet replica
I20260806 03:44:45.435174 12953 tablet_replica.cc:330] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: stopping tablet replica
I20260806 03:44:45.435279 12953 raft_consensus.cc:2229] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:44:45.435317 12954 raft_consensus.cc:2229] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:44:45.435446 12954 raft_consensus.cc:2258] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:45.435480 12953 raft_consensus.cc:2258] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:45.435973 12954 ts_tablet_manager.cc:1905] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:45.436025 12953 ts_tablet_manager.cc:1905] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:45.437774 12954 ts_tablet_manager.cc:1918] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:45.437932 12954 log.cc:1196] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/dfba77cca6ea4155bb512d7edde43238
I20260806 03:44:45.438289 12953 ts_tablet_manager.cc:1918] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:45.438328 12954 ts_tablet_manager.cc:1939] T dfba77cca6ea4155bb512d7edde43238 P 7103a0b492924f59932fb88796259587: Deleting consensus metadata
I20260806 03:44:45.438452 12953 log.cc:1196] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/dfba77cca6ea4155bb512d7edde43238
I20260806 03:44:45.438753 12421 catalog_manager.cc:4841] TS 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): tablet dfba77cca6ea4155bb512d7edde43238 (table test-table [id=b2f61c79383d46e395c49128aed64419]) successfully deleted
I20260806 03:44:45.438804 12953 ts_tablet_manager.cc:1939] T dfba77cca6ea4155bb512d7edde43238 P c45122be1aec43ecaa5491ce834263d7: Deleting consensus metadata
I20260806 03:44:45.439221 12423 catalog_manager.cc:4841] TS c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): tablet dfba77cca6ea4155bb512d7edde43238 (table test-table [id=b2f61c79383d46e395c49128aed64419]) successfully deleted
W20260806 03:44:45.442744 12421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:42217 (error 108)
I20260806 03:44:45.442937 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12639
I20260806 03:44:45.445807 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.445646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:42217 (local address 127.0.0.1:54494)
0806 03:44:45.445713 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.445767 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:42217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 03:44:45.445945 12421 catalog_manager.cc:4584] TS a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfba77cca6ea4155bb512d7edde43238: Network error: Client connection negotiation failed: client connection to 127.30.24.1:42217: connect: Connection refused (error 111)
I20260806 03:44:45.449559 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12777
I20260806 03:44:45.454777 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12404
I20260806 03:44:45.460893 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34817
--webserver_interface=127.30.24.62
--webserver_port=38013
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:34817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:45.544224 12956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:45.544512 12956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:45.544553 12956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:45.545738 12956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:45.545804 12956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:45.545828 12956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:45.545850 12956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:45.545878 12956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:45.547407 12956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:34817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=38013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:45.547696 12956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:45.547940 12956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:45.557485 12962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.557489 12963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.557485 12965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:45.557787 12956 server_base.cc:1034] running on GCE node
I20260806 03:44:45.558048 12956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:45.558439 12956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:45.559615 12956 hybrid_clock.cc:648] HybridClock initialized: now 1785987885559602 us; error 37 us; skew 500 ppm
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:45.562340 12956 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:45.562812 12956 webserver.cc:460] Webserver started at http://127.30.24.62:38013/ using document root <none> and password file <none>
I20260806 03:44:45.563067 12956 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:45.563165 12956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:45.564671 12956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:45.565388 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.565599 12956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:45.565711 12956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "dd0093b7a5df43e2901e592b4f575852"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "6185ce5d5210452ff3a2afd880e13848"
server_key_iv: "80bb5116e34777a07ea4841f644b2e45"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.566165 12956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:45.595752 12956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:45.597765 12956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:45.597934 12956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:45.602069 12956 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:34817
I20260806 03:44:45.602102 13025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:34817 every 8 connection(s)
I20260806 03:44:45.602530 12956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:45.604220 13026 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:44:45.604911 13026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap starting.
I20260806 03:44:45.606187 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 12956
I20260806 03:44:45.606510 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:42217
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:34817
--webserver_port=46253
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:45.614425 13026 log.cc:824] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:45.616418 13026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:44:45.616786 13026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap complete.
I20260806 03:44:45.619127 13026 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:45.619366 13026 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0093b7a5df43e2901e592b4f575852, State: Initialized, Role: FOLLOWER
I20260806 03:44:45.619526 13026 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:45.619594 13026 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:44:45.619644 13026 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:44:45.619694 13026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:44:45.620764 13026 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:45.620872 13026 leader_election.cc:302] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0093b7a5df43e2901e592b4f575852; no voters:
I20260806 03:44:45.621066 13026 leader_election.cc:288] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:44:45.621121 13030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:44:45.621289 13030 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 LEADER]: Becoming Leader. State: Replica: dd0093b7a5df43e2901e592b4f575852, State: Running, Role: LEADER
I20260806 03:44:45.621454 13026 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:45.621444 13030 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:45.621915 13030 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0093b7a5df43e2901e592b4f575852. Latest consensus state: current_term: 2 leader_uuid: "dd0093b7a5df43e2901e592b4f575852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } } }
I20260806 03:44:45.622159 13030 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:45.622095 13031 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dd0093b7a5df43e2901e592b4f575852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } } }
I20260806 03:44:45.622489 13031 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:45.622612 13038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:45.623467 13038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:44:45.623623 13038 catalog_manager.cc:1259] Loaded cluster ID: a9e731f91b894e53a47b505ed561b48a
I20260806 03:44:45.623898 13038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:44:45.624164 13038 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 03:44:45.629205 13038 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 03:44:45.631723 13038 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:44:45.631930 13038 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Loaded TSK: 0
I20260806 03:44:45.632150 13038 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:44:45.695978 13028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:45.696184 13028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:45.696208 13028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:45.697445 13028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:45.697490 13028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:45.697517 13028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:44:45.698990 13028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:42217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=46253
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:45.699250 13028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:45.699498 13028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:45.704557 13057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.704619 13053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.704566 13054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:45.705189 13028 server_base.cc:1034] running on GCE node
I20260806 03:44:45.705417 13028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:45.705693 13028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:45.706877 13028 hybrid_clock.cc:648] HybridClock initialized: now 1785987885706862 us; error 38 us; skew 500 ppm
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:45.709789 13028 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:44:45.710260 13028 webserver.cc:460] Webserver started at http://127.30.24.1:46253/ using document root <none> and password file <none>
I20260806 03:44:45.710501 13028 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:45.710557 13028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:45.711995 13028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:45.712728 13067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.712920 13028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:45.713016 13028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "a69d5f7783254b9c9ab30dd72336db9e"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "7c0ea289514625a817a3d337b6fe7f1c"
server_key_iv: "fffc8b3b12889b5e7c4ea4984fdd3779"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.713408 13028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:45.737279 13028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:45.739729 13028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:45.739924 13028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:45.740240 13028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:45.740804 13074 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 03:44:45.742651 13028 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 03:44:45.742733 13028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 03:44:45.742774 13028 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 03:44:45.743462 13028 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 03:44:45.743520 13028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 03:44:45.743572 13074 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap starting.
I20260806 03:44:45.750892 13028 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:42217
I20260806 03:44:45.750963 13188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:42217 every 8 connection(s)
I20260806 03:44:45.751302 13028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:44:45.753034 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13028
I20260806 03:44:45.753328 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:45081
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:34817
--webserver_port=45897
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:45.755097 13074 log.cc:824] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:45.757443 13074 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:44:45.757735 13074 tablet_bootstrap.cc:492] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap complete.
I20260806 03:44:45.758857 13074 ts_tablet_manager.cc:1397] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:45.760313 13074 raft_consensus.cc:348] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.760706 13074 raft_consensus.cc:729] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d5f7783254b9c9ab30dd72336db9e, State: Initialized, Role: FOLLOWER
I20260806 03:44:45.760848 13074 consensus_queue.cc:260] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:45.761137 13074 ts_tablet_manager.cc:1428] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260806 03:44:45.765074 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.752038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:34735)
0806 03:44:45.752249 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.752293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.753064 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.753821 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.753825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.754352 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:45.754681 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.754688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.756105 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.756106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.757448 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.757451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.758340 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.758341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.758500 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.759132 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:45.759142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:45.760967 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:45.763419 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.763426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.763428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.763653 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.763964 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.763965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.763965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.764069 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.764585 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:45.764592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:45.764711 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.764796 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.764920 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":126,"threads_started":1}
I20260806 03:44:45.765292 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.752147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:34735 (local address 127.30.24.62:34817)
0806 03:44:45.752322 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.752331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.752391 (+ 60us) server_negotiation.cc:407] Connection header received
0806 03:44:45.753384 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.753394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.753472 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.753623 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:45.754822 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.755971 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.757551 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.758206 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.758323 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.761087 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:45.761092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:45.761092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:45.761106 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:45.763339 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.763764 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.763767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.763768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.763895 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.764145 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.764146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.764147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.764510 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:45.764608 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.764906 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.765082 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260806 03:44:45.765909 13190 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34817
I20260806 03:44:45.766019 13190 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:45.766228 13190 heartbeater.cc:511] Master 127.30.24.62:34817 requested a full tablet report, sending...
I20260806 03:44:45.766980 12989 ts_manager.cc:194] Registered new tserver with Master: a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217)
I20260806 03:44:45.768117 12989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:34735
I20260806 03:44:45.769065 13190 heartbeater.cc:503] Master 127.30.24.62:34817 was elected leader, sending a full tablet report...
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:44:45.776729 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.767581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:42217 (local address 127.0.0.1:54500)
0806 03:44:45.767763 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.767798 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.767864 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.768050 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.768054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.768371 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:45.768430 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.768432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.770485 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.770486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.771065 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.771068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.771671 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.771672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.771734 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.772277 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:45.772281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:45.773633 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:45.775572 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.775579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.775581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.775796 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.776015 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.776016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.776016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.776074 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.776371 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:45.776377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:45.776420 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.776544 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.776658 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260806 03:44:45.776796 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.767670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.30.24.1:42217)
0806 03:44:45.767830 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.767834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.767845 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:44:45.767954 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.767956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.767973 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.768036 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:45.768519 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.770411 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.771145 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.771589 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.771664 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.773713 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:45.773721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:45.773722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:45.773729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:45.775477 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.775880 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.775883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.775883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.775948 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.776130 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.776131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.776131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.776307 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:45.776390 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.776593 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.776698 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260806 03:44:45.777148 13119 tablet_service.cc:1514] Processing DeleteTablet for tablet dfba77cca6ea4155bb512d7edde43238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:45 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:54500
I20260806 03:44:45.777707 13199 tablet_replica.cc:330] stopping tablet replica
I20260806 03:44:45.777824 13199 raft_consensus.cc:2229] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:44:45.777917 13199 raft_consensus.cc:2258] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:45.778220 13199 ts_tablet_manager.cc:1905] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:45.780371 13199 ts_tablet_manager.cc:1918] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:45.780507 13199 log.cc:1196] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/dfba77cca6ea4155bb512d7edde43238
I20260806 03:44:45.780766 13199 ts_tablet_manager.cc:1939] T dfba77cca6ea4155bb512d7edde43238 P a69d5f7783254b9c9ab30dd72336db9e: Deleting consensus metadata
I20260806 03:44:45.781293 12974 catalog_manager.cc:4841] TS a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217): tablet dfba77cca6ea4155bb512d7edde43238 (table test-table [id=b2f61c79383d46e395c49128aed64419]) successfully deleted
W20260806 03:44:45.843284 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:45.843502 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:45.843542 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:45.844741 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:45.844799 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:45.844844 13194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:44:45.846287 13194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:45081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=45897
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:45.846551 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:45.846786 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:45.851755 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.851845 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.851857 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:45.852015 13194 server_base.cc:1034] running on GCE node
I20260806 03:44:45.852228 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:45.852422 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:45.853605 13194 hybrid_clock.cc:648] HybridClock initialized: now 1785987885853590 us; error 33 us; skew 500 ppm
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:45.855967 13194 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:44:45.856426 13194 webserver.cc:460] Webserver started at http://127.30.24.2:45897/ using document root <none> and password file <none>
I20260806 03:44:45.856664 13194 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:45.856746 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:45.858050 13194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:44:45.858590 13214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.858757 13194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:44:45.858851 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "c45122be1aec43ecaa5491ce834263d7"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "95d73a458f820352b1c2dceaf4cd309d"
server_key_iv: "3697bc413f5701e3835851e7188089a0"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.859193 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:45.871734 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:45.873591 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:45.873756 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:45.873972 13194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:45.874281 13194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:45.874330 13194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.874369 13194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:45.874396 13194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.880214 13194 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:45081
I20260806 03:44:45.880266 13333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:45081 every 8 connection(s)
I20260806 03:44:45.880661 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:45.889730 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.881362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:53329 (local address 127.30.24.62:34817)
0806 03:44:45.881417 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:45.881419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:45.881602 (+ 183us) server_negotiation.cc:407] Connection header received
0806 03:44:45.882180 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:45.882182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:45.882196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:45.882248 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:45.883225 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.883725 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.884761 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.884972 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.885060 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.887151 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:45.887152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:45.887153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:45.887160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:45.888373 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.888787 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.888789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.888789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.888851 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:45.889089 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:45.889090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:45.889090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:45.889212 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:45.889286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:45.889545 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 03:44:45.889636 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:44:45.889910 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:45.881331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:53329)
0806 03:44:45.881474 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:45.881515 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:45.882033 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:45.882328 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:45.882332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:45.882918 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:45.883105 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.883110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.883827 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.883829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.884667 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:45.884670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:45.885069 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:45.885070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:45.885196 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:45.885690 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:45.885699 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:45.887052 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:45.888484 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.888491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.888494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.888711 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.888916 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:45.888917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:45.888918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:45.889025 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:45.889278 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:45.889285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:45.889419 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 03:44:45.889521 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 03:44:45.889681 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260806 03:44:45.890322 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13194
I20260806 03:44:45.890353 13334 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34817
I20260806 03:44:45.890450 13334 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:45.890618 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:37579
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:34817
--webserver_port=43575
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:45.890632 13334 heartbeater.cc:511] Master 127.30.24.62:34817 requested a full tablet report, sending...
I20260806 03:44:45.891039 12989 ts_manager.cc:194] Registered new tserver with Master: c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081)
I20260806 03:44:45.891417 12989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:53329
W20260806 03:44:45.974268 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:45.974510 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:45.974547 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:45.975698 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:45.975759 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:45.975805 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:44:45.977317 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:37579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=43575
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:45.977571 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:45.977798 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:45.982607 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.982652 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:45.982652 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:45.982995 13337 server_base.cc:1034] running on GCE node
I20260806 03:44:45.983212 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:45.983444 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:45.984596 13337 hybrid_clock.cc:648] HybridClock initialized: now 1785987885984589 us; error 35 us; skew 500 ppm
Aug 06 03:44:45 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:45.987093 13337 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:44:45.987555 13337 webserver.cc:460] Webserver started at http://127.30.24.3:43575/ using document root <none> and password file <none>
I20260806 03:44:45.987777 13337 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:45.987851 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:45.989264 13337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:45.989820 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:45.989980 13337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:44:45.990074 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "7103a0b492924f59932fb88796259587"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "513078dd71b807ada4f8bc698497c941"
server_key_iv: "dc0305a8292cb89bd5113ac686b9558a"
server_key_version: "encryptionkey@0"
I20260806 03:44:45.990419 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:46.009421 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:46.011842 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:46.012010 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:46.012267 13337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:46.012605 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:46.012666 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.012727 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:46.012759 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.018941 13337 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37579
I20260806 03:44:46.018990 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37579 every 8 connection(s)
I20260806 03:44:46.019341 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:44:46 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987885, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:44:46.026408 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13337
I20260806 03:44:46.028884 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.020049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:60175 (local address 127.30.24.62:34817)
0806 03:44:46.020098 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.020100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.020320 (+ 220us) server_negotiation.cc:407] Connection header received
0806 03:44:46.020891 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.020892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.020902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.020975 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:46.022010 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.022529 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.023538 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.023727 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.023781 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.025925 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:46.025926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:46.025926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:46.025933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:46.027523 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:46.027944 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:46.027945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:46.027945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:46.028008 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:46.028236 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:46.028237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:46.028237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:46.028359 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:46.028425 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.028752 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.028820 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:44:46.029124 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.020014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:60175)
0806 03:44:46.020191 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.020235 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:46.020802 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:46.021087 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:46.021090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:46.021729 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:46.021931 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.021936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.022609 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.022610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.023465 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.023467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.023798 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.023799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.023899 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.024360 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:46.024369 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:46.025828 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:46.027598 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:46.027607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:46.027610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:46.027879 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:46.028076 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:46.028077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:46.028077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:46.028177 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:46.028424 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:46.028430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:46.028608 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 03:44:46.028736 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 03:44:46.028898 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260806 03:44:46.029682 13470 heartbeater.cc:347] Connected to a master server at 127.30.24.62:34817
I20260806 03:44:46.029793 13470 heartbeater.cc:464] Registering TS with master...
I20260806 03:44:46.029996 13470 heartbeater.cc:511] Master 127.30.24.62:34817 requested a full tablet report, sending...
I20260806 03:44:46.030469 12989 ts_manager.cc:194] Registered new tserver with Master: 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
I20260806 03:44:46.031008 12989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:60175
I20260806 03:44:46.033396 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.026808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.30.24.62:34817)
0806 03:44:46.027025 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.027028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.027037 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:44:46.027061 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.027062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.027075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.027152 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:46.027705 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.028345 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.029379 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.029709 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.029781 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.031132 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:46.031133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:46.031134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:46.031141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:46.032441 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:46.032782 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:46.032783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:46.032783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:46.032834 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:46.032992 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:46.032993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:46.032993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:46.033094 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:46.033219 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.033274 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.033335 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":61,"threads_started":1}
I20260806 03:44:46.034068 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:44:46.036571 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.034408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.30.24.62:34817)
0806 03:44:46.034446 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.034448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.034573 (+ 125us) server_negotiation.cc:407] Connection header received
0806 03:44:46.034642 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.034642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.034671 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.034725 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:44:46.034808 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.035243 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.035930 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.036106 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.036162 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.036274 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:44:46.036362 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:44:46.036415 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.036443 (+ 28us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.036499 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:44:46.036859 12989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:44:46.038522 12989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:44:46.040977 13119 tablet_service.cc:1467] Processing CreateTablet for tablet d0391f35acc948f0ab4c6eba205d48e5 (DEFAULT_TABLE table=test-table [id=1062af563ae54ae48dfb0a8b595b7db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:46.041201 13119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0391f35acc948f0ab4c6eba205d48e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:46.042871 13074 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap starting.
I20260806 03:44:46.043257 13074 tablet_bootstrap.cc:654] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:46.043910 13074 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: No bootstrap required, opened a new log
I20260806 03:44:46.043970 13074 ts_tablet_manager.cc:1397] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260806 03:44:46.044348 13074 raft_consensus.cc:348] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.044436 13074 raft_consensus.cc:374] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:46.044518 13074 raft_consensus.cc:729] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d5f7783254b9c9ab30dd72336db9e, State: Initialized, Role: FOLLOWER
I20260806 03:44:46.044567 13074 consensus_queue.cc:260] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.044724 13074 ts_tablet_manager.cc:1428] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:46.045482 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.040567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37579 (local address 127.0.0.1:44066)
0806 03:44:46.040732 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.040744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:46.040793 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:46.041305 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:46.041307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:46.041310 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:46.041354 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.041356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.042613 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.042614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.043872 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.043875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.045189 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.045190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.045278 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.045303 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 03:44:46.045354 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 03:44:46.045398 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260806 03:44:46.045753 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.040850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.30.24.3:37579)
0806 03:44:46.041052 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.041057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.041102 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:44:46.041144 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.041147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.041165 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.041251 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:46.041430 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.042491 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.044026 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.044935 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.045015 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.045099 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.045625 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.045670 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260806 03:44:46.046427 13401 tablet_service.cc:1467] Processing CreateTablet for tablet d0391f35acc948f0ab4c6eba205d48e5 (DEFAULT_TABLE table=test-table [id=1062af563ae54ae48dfb0a8b595b7db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:46.046815 13401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0391f35acc948f0ab4c6eba205d48e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:46.046834 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.040589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.30.24.2:45081)
0806 03:44:46.043039 (+ 2450us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.043045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.043056 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:44:46.043099 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.043101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.043118 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.043210 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:46.043346 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.044701 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.045968 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.046472 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.046529 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.046573 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.046738 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.046772 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2398,"thread_start_us":122,"threads_started":1}
I20260806 03:44:46.046841 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.040517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:45081 (local address 127.0.0.1:34608)
0806 03:44:46.040557 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.040583 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:46.040625 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:46.043218 (+ 2593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:46.043220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:46.043222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:46.043260 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.043262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.044847 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.044848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.045858 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.045861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.046567 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.046568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.046643 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.046648 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:44:46.046716 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 03:44:46.046769 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 03:44:46.047551 13264 tablet_service.cc:1467] Processing CreateTablet for tablet d0391f35acc948f0ab4c6eba205d48e5 (DEFAULT_TABLE table=test-table [id=1062af563ae54ae48dfb0a8b595b7db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:44:46.047883 13264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0391f35acc948f0ab4c6eba205d48e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:44:46.050642 13480 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Bootstrap starting.
I20260806 03:44:46.051311 13481 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Bootstrap starting.
I20260806 03:44:46.051344 13480 tablet_bootstrap.cc:654] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:46.051927 13480 log.cc:824] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:46.052024 13481 tablet_bootstrap.cc:654] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Neither blocks nor log segments found. Creating new log.
I20260806 03:44:46.052592 13481 log.cc:824] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:46.052788 13480 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: No bootstrap required, opened a new log
I20260806 03:44:46.052870 13480 ts_tablet_manager.cc:1397] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:44:46.053402 13481 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: No bootstrap required, opened a new log
I20260806 03:44:46.053475 13481 ts_tablet_manager.cc:1397] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:44:46.054924 13480 raft_consensus.cc:348] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.055101 13480 raft_consensus.cc:374] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:46.055137 13480 raft_consensus.cc:729] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7103a0b492924f59932fb88796259587, State: Initialized, Role: FOLLOWER
I20260806 03:44:46.055276 13480 consensus_queue.cc:260] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.055563 13480 ts_tablet_manager.cc:1428] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 03:44:46.055464 13481 raft_consensus.cc:348] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.055608 13481 raft_consensus.cc:374] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:44:46.055611 13470 heartbeater.cc:503] Master 127.30.24.62:34817 was elected leader, sending a full tablet report...
I20260806 03:44:46.055650 13481 raft_consensus.cc:729] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45122be1aec43ecaa5491ce834263d7, State: Initialized, Role: FOLLOWER
I20260806 03:44:46.055774 13481 consensus_queue.cc:260] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.056068 13481 ts_tablet_manager.cc:1428] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:44:46.056080 13334 heartbeater.cc:503] Master 127.30.24.62:34817 was elected leader, sending a full tablet report...
I20260806 03:44:46.111331 13486 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:46.111517 13486 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.111989 13486 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7103a0b492924f59932fb88796259587 (127.30.24.3:37579), a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217)
I20260806 03:44:46.115913 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.112410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:36779 (local address 127.30.24.1:42217)
0806 03:44:46.112486 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.112488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.112605 (+ 117us) server_negotiation.cc:407] Connection header received
0806 03:44:46.112651 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.112652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.112675 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.112723 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:46.112871 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.113587 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.115123 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.115609 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.115676 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.115735 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.115829 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.115860 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 03:44:46.115953 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.112315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:42217 (local address 127.30.24.2:36779)
0806 03:44:46.112513 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.112526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:46.112594 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:46.112744 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:46.112745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:46.112746 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:46.112810 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.112811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.113656 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.113657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.115037 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.115040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.115682 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.115683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.115756 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.115761 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:44:46.115824 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 03:44:46.115865 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":57,"thread_start_us":106,"threads_started":1}
I20260806 03:44:46.116524 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.112201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37579 (local address 127.30.24.2:37603)
0806 03:44:46.112285 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.112300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:46.112369 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:46.112578 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:46.112579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:46.112582 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:44:46.112682 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.112684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.113638 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.113639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.115037 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:46.115040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.116212 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.116213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:46.116331 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.116335 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:44:46.116397 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:44:46.116435 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":78}
I20260806 03:44:46.116547 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.112346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:37603 (local address 127.30.24.3:37579)
0806 03:44:46.112399 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.112401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.112412 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:44:46.112447 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.112448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.112488 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.112550 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:44:46.112771 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.113522 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.115517 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.116123 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.116213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.116224 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.116409 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.116444 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:44:46.116436 13139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0391f35acc948f0ab4c6eba205d48e5" candidate_uuid: "c45122be1aec43ecaa5491ce834263d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d5f7783254b9c9ab30dd72336db9e" is_pre_election: true
I20260806 03:44:46.116674 13139 raft_consensus.cc:2454] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45122be1aec43ecaa5491ce834263d7 in term 0.
I20260806 03:44:46.116963 13216 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e, c45122be1aec43ecaa5491ce834263d7; no voters:
I20260806 03:44:46.117101 13486 raft_consensus.cc:2781] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:44:46.117162 13486 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:46.117195 13486 raft_consensus.cc:3037] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:46.117112 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0391f35acc948f0ab4c6eba205d48e5" candidate_uuid: "c45122be1aec43ecaa5491ce834263d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7103a0b492924f59932fb88796259587" is_pre_election: true
I20260806 03:44:46.117278 13422 raft_consensus.cc:2454] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45122be1aec43ecaa5491ce834263d7 in term 0.
I20260806 03:44:46.117985 13486 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.118132 13486 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7103a0b492924f59932fb88796259587 (127.30.24.3:37579), a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217)
I20260806 03:44:46.118273 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0391f35acc948f0ab4c6eba205d48e5" candidate_uuid: "c45122be1aec43ecaa5491ce834263d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7103a0b492924f59932fb88796259587"
I20260806 03:44:46.118323 13139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0391f35acc948f0ab4c6eba205d48e5" candidate_uuid: "c45122be1aec43ecaa5491ce834263d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d5f7783254b9c9ab30dd72336db9e"
I20260806 03:44:46.118363 13422 raft_consensus.cc:3037] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:46.118424 13139 raft_consensus.cc:3037] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:44:46.119081 13422 raft_consensus.cc:2454] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45122be1aec43ecaa5491ce834263d7 in term 1.
I20260806 03:44:46.119167 13139 raft_consensus.cc:2454] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45122be1aec43ecaa5491ce834263d7 in term 1.
I20260806 03:44:46.119364 13216 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e, c45122be1aec43ecaa5491ce834263d7; no voters:
I20260806 03:44:46.119467 13486 raft_consensus.cc:2781] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:44:46.119614 13486 raft_consensus.cc:686] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 1 LEADER]: Becoming Leader. State: Replica: c45122be1aec43ecaa5491ce834263d7, State: Running, Role: LEADER
I20260806 03:44:46.119715 13486 consensus_queue.cc:237] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.120609 12989 catalog_manager.cc:5495] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c45122be1aec43ecaa5491ce834263d7 (127.30.24.2). New cstate: current_term: 1 leader_uuid: "c45122be1aec43ecaa5491ce834263d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } health_report { overall_health: UNKNOWN } } }
W20260806 03:44:46.131218 13335 tablet.cc:2369] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:44:46.137802 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.135642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.30.24.2:45081)
0806 03:44:46.135729 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:46.135731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:46.135772 (+ 41us) server_negotiation.cc:407] Connection header received
0806 03:44:46.135853 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:46.135854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:46.135874 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:46.135927 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:44:46.136018 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.136548 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.137214 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:46.137399 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:46.137446 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:46.137553 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:44:46.137642 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:44:46.137693 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:46.137720 (+ 27us) server_negotiation.cc:299] Negotiation successful
0806 03:44:46.137734 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:44:46.139604 13139 raft_consensus.cc:1261] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Refusing update from remote peer c45122be1aec43ecaa5491ce834263d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:46.139617 13422 raft_consensus.cc:1261] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Refusing update from remote peer c45122be1aec43ecaa5491ce834263d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:44:46.139997 13488 consensus_queue.cc:1035] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:46.140158 13486 consensus_queue.cc:1035] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:44:46.142488 13492 mvcc.cc:205] Tried to move back new op lower bound from 7315406381624954880 to 7315406381547102208. Current Snapshot: MvccSnapshot[applied={T|T < 7315406381624954880}]
I20260806 03:44:46.142941 13493 mvcc.cc:205] Tried to move back new op lower bound from 7315406381624954880 to 7315406381547102208. Current Snapshot: MvccSnapshot[applied={T|T < 7315406381624954880}]
I20260806 03:44:46.144707 12989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:44:46.144810 12989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:44:46.145532 12989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Sending DeleteTablet for 3 replicas of tablet d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:46.145843 13264 tablet_service.cc:1514] Processing DeleteTablet for tablet d0391f35acc948f0ab4c6eba205d48e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:46 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:34608
I20260806 03:44:46.145859 13119 tablet_service.cc:1514] Processing DeleteTablet for tablet d0391f35acc948f0ab4c6eba205d48e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:46 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:54500
I20260806 03:44:46.145874 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet d0391f35acc948f0ab4c6eba205d48e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:44:46 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:44066
I20260806 03:44:46.145996 13199 tablet_replica.cc:330] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: stopping tablet replica
I20260806 03:44:46.146068 13199 raft_consensus.cc:2229] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:44:46.146127 13199 raft_consensus.cc:2258] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:46.146186 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 13028
I20260806 03:44:46.146281 13501 tablet_replica.cc:330] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: stopping tablet replica
I20260806 03:44:46.146371 13501 raft_consensus.cc:2229] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:44:46.146452 13501 raft_consensus.cc:2258] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:46.149138 13501 ts_tablet_manager.cc:1905] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:46.146100 13500 tablet_replica.cc:330] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: stopping tablet replica
I20260806 03:44:46.149631 13500 raft_consensus.cc:2229] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:44:46.149797 13500 raft_consensus.cc:2258] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:44:46.150288 13500 ts_tablet_manager.cc:1905] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:44:46.150935 13501 ts_tablet_manager.cc:1918] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:46.151017 13501 log.cc:1196] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:46.151356 13501 ts_tablet_manager.cc:1939] T d0391f35acc948f0ab4c6eba205d48e5 P c45122be1aec43ecaa5491ce834263d7: Deleting consensus metadata
I20260806 03:44:46.151808 12976 catalog_manager.cc:4841] TS c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): tablet d0391f35acc948f0ab4c6eba205d48e5 (table test-table [id=1062af563ae54ae48dfb0a8b595b7db8]) successfully deleted
I20260806 03:44:46.152256 13500 ts_tablet_manager.cc:1918] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:44:46.152360 13500 log.cc:1196] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:46.152683 13500 ts_tablet_manager.cc:1939] T d0391f35acc948f0ab4c6eba205d48e5 P 7103a0b492924f59932fb88796259587: Deleting consensus metadata
I20260806 03:44:46.153091 12974 catalog_manager.cc:4841] TS 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): tablet d0391f35acc948f0ab4c6eba205d48e5 (table test-table [id=1062af563ae54ae48dfb0a8b595b7db8]) successfully deleted
W20260806 03:44:46.155224 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:42217 (error 108)
I20260806 03:44:46.155417 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 13194
I20260806 03:44:46.160084 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.159901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:42217 (local address 127.0.0.1:54502)
0806 03:44:46.159972 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.160047 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:42217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 03:44:46.160233 12974 catalog_manager.cc:4584] TS a69d5f7783254b9c9ab30dd72336db9e (127.30.24.1:42217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0391f35acc948f0ab4c6eba205d48e5: Network error: Client connection negotiation failed: client connection to 127.30.24.1:42217: connect: Connection refused (error 111)
I20260806 03:44:46.164121 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 13337
I20260806 03:44:46.169896 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 12956
I20260806 03:44:46.176141 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34817
--webserver_interface=127.30.24.62
--webserver_port=38013
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:34817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:46.265827 13503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:46.266042 13503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:46.266067 13503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:46.267160 13503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:44:46.267194 13503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:46.267210 13503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:46.267222 13503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:44:46.267234 13503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:44:46.268754 13503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:34817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:34817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=38013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:46.268966 13503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:46.269156 13503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:46.274180 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:46.274180 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:46.274268 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:46.274435 13503 server_base.cc:1034] running on GCE node
I20260806 03:44:46.274689 13503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:46.274984 13503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:46.276170 13503 hybrid_clock.cc:648] HybridClock initialized: now 1785987886276142 us; error 34 us; skew 500 ppm
Aug 06 03:44:46 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987886, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:46.278836 13503 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:44:46.279304 13503 webserver.cc:460] Webserver started at http://127.30.24.62:38013/ using document root <none> and password file <none>
I20260806 03:44:46.279548 13503 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:46.279625 13503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:46.281062 13503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:44:46.281782 13518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.281949 13503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:44:46.282045 13503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "dd0093b7a5df43e2901e592b4f575852"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "6185ce5d5210452ff3a2afd880e13848"
server_key_iv: "80bb5116e34777a07ea4841f644b2e45"
server_key_version: "encryptionkey@0"
I20260806 03:44:46.282401 13503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:46.294293 13503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:46.296145 13503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:46.296303 13503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:46.300241 13503 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:34817
I20260806 03:44:46.300288 13571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:34817 every 8 connection(s)
I20260806 03:44:46.300748 13503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:44:46.301362 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13503
I20260806 03:44:46.301728 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:42217
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:34817
--webserver_port=46253
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:46.303139 13572 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:44:46.304013 13572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap starting.
I20260806 03:44:46.312647 13572 log.cc:824] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:46.314952 13572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:44:46.315287 13572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852: Bootstrap complete.
I20260806 03:44:46.317754 13572 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:46.318003 13572 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0093b7a5df43e2901e592b4f575852, State: Initialized, Role: FOLLOWER
I20260806 03:44:46.318197 13572 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:46.318289 13572 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:44:46.318346 13572 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:44:46.318393 13572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:44:46.319355 13572 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:46.319473 13572 leader_election.cc:302] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0093b7a5df43e2901e592b4f575852; no voters:
I20260806 03:44:46.319670 13572 leader_election.cc:288] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 03:44:46.319708 13576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:44:46.319821 13576 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [term 3 LEADER]: Becoming Leader. State: Replica: dd0093b7a5df43e2901e592b4f575852, State: Running, Role: LEADER
I20260806 03:44:46.319918 13576 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } }
I20260806 03:44:46.319945 13572 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:44:46.320194 13577 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dd0093b7a5df43e2901e592b4f575852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } } }
I20260806 03:44:46.320341 13577 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:46.320194 13576 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0093b7a5df43e2901e592b4f575852. Latest consensus state: current_term: 3 leader_uuid: "dd0093b7a5df43e2901e592b4f575852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0093b7a5df43e2901e592b4f575852" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 34817 } } }
I20260806 03:44:46.320441 13576 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0093b7a5df43e2901e592b4f575852 [sys.catalog]: This master's current role is: LEADER
I20260806 03:44:46.320720 13584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:44:46.321501 13584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:44:46.321614 13584 catalog_manager.cc:1259] Loaded cluster ID: a9e731f91b894e53a47b505ed561b48a
I20260806 03:44:46.321767 13584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 03:44:46.322357 13584 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0d73bb59ad google::LogMessage::Fail() at ??:0
@ 0x7f0d73bb888a google::LogMessage::SendToLog() at ??:0
@ 0x7f0d73bb54de google::LogMessage::Flush() at ??:0
@ 0x7f0d73bb6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0d7bad60f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0d7badf8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0d75a830d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0d75a78954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0d768dc6db start_thread at ??:0
@ 0x7f0d726b361f clone at ??:0
W20260806 03:44:46.402633 13574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:46.402827 13574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:46.402849 13574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:46.404031 13574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:46.404078 13574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:46.404104 13574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:44:46.405642 13574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:42217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=46253
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:46.405879 13574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:46.406128 13574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:46.411197 13612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:46.411202 13609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:46.411206 13610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:46.411419 13574 server_base.cc:1034] running on GCE node
I20260806 03:44:46.411813 13574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:46.412073 13574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:46.413259 13574 hybrid_clock.cc:648] HybridClock initialized: now 1785987886413236 us; error 37 us; skew 500 ppm
Aug 06 03:44:46 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987886, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:46.416296 13574 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:44:46.416848 13574 webserver.cc:460] Webserver started at http://127.30.24.1:46253/ using document root <none> and password file <none>
I20260806 03:44:46.417099 13574 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:46.417169 13574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:46.418565 13574 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:46.419324 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.419541 13574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:46.419641 13574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "a69d5f7783254b9c9ab30dd72336db9e"
format_stamp: "Formatted at 2026-08-06 03:44:44 on dist-test-slave-10pc"
server_key: "7c0ea289514625a817a3d337b6fe7f1c"
server_key_iv: "fffc8b3b12889b5e7c4ea4984fdd3779"
server_key_version: "encryptionkey@0"
I20260806 03:44:46.419998 13574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:46.440034 13574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:46.442541 13574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:46.442719 13574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:46.442982 13574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:46.443468 13626 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 03:44:46.445565 13574 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 03:44:46.445631 13574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 03:44:46.445675 13574 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 03:44:46.446346 13574 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 03:44:46.446399 13574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 03:44:46.446478 13626 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap starting.
I20260806 03:44:46.453828 13574 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:42217
I20260806 03:44:46.453896 13735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:42217 every 8 connection(s)
I20260806 03:44:46.454306 13574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:44:46.457916 13626 log.cc:824] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:44:46.458240 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13574
I20260806 03:44:46.458559 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:45081
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:34817
--webserver_port=45897
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:44:46.460795 13626 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:44:46.461112 13626 tablet_bootstrap.cc:492] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Bootstrap complete.
I20260806 03:44:46.462395 13626 ts_tablet_manager.cc:1397] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260806 03:44:46.463881 13626 raft_consensus.cc:348] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.464237 13626 raft_consensus.cc:729] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d5f7783254b9c9ab30dd72336db9e, State: Initialized, Role: FOLLOWER
I20260806 03:44:46.464437 13626 consensus_queue.cc:260] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:46.464792 13626 ts_tablet_manager.cc:1428] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260806 03:44:46.514284 13738 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:44:46.455052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:51849)
0806 03:44:46.455235 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.455279 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:46.455742 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:46.514081 (+ 58339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260806 03:44:46.514580 13736 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:34817 (0 consecutive failures): Network error: Failed to ping master at 127.30.24.62:34817: Client connection negotiation failed: client connection to 127.30.24.62:34817: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 03:44:46.514991 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.514851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:41373)
0806 03:44:46.514927 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.514971 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 03:44:46.515347 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.515255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:54683)
0806 03:44:46.515299 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.515328 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 03:44:46.515455 13736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:44:46.546484 13740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:46.546700 13740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:46.546725 13740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:46.547868 13740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:46.547911 13740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:46.547940 13740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:44:46.549491 13740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:45081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=45897
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:46.549755 13740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:46.549986 13740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:46.554999 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:46.555018 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:46.555198 13740 server_base.cc:1034] running on GCE node
W20260806 03:44:46.555013 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:46.555475 13740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:46.555729 13740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:46.556892 13740 hybrid_clock.cc:648] HybridClock initialized: now 1785987886556879 us; error 40 us; skew 500 ppm
Aug 06 03:44:46 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987886, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:46.559340 13740 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:44:46.559809 13740 webserver.cc:460] Webserver started at http://127.30.24.2:45897/ using document root <none> and password file <none>
I20260806 03:44:46.560042 13740 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:46.560109 13740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:46.561575 13740 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:46.562191 13756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.562390 13740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:44:46.562491 13740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "c45122be1aec43ecaa5491ce834263d7"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "95d73a458f820352b1c2dceaf4cd309d"
server_key_iv: "3697bc413f5701e3835851e7188089a0"
server_key_version: "encryptionkey@0"
I20260806 03:44:46.562868 13740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:46.572044 13740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:46.574126 13740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:46.574291 13740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:46.574524 13740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:46.574852 13740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:46.574905 13740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.574939 13740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:46.574966 13740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.581076 13740 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:45081
I20260806 03:44:46.581130 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:45081 every 8 connection(s)
I20260806 03:44:46.581487 13740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:44:46.582520 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.582144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:34641)
0806 03:44:46.582292 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.582374 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
W20260806 03:44:46.582942 13872 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:34817 (0 consecutive failures): Network error: Failed to ping master at 127.30.24.62:34817: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
I20260806 03:44:46.583235 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.583147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:41035)
0806 03:44:46.583190 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.583217 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 03:44:46.583597 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.583529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:44245)
0806 03:44:46.583562 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.583583 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 03:44:46.583725 13872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:44:46.585613 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13740
I20260806 03:44:46.585876 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:37579
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:34817
--webserver_port=43575
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:44:46.670109 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:44:46.670323 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:44:46.670347 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:44:46.671424 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:44:46.671458 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:44:46.671484 13875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:44:46.672858 13875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:37579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=43575
--tserver_master_addrs=127.30.24.62:34817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:44:46.673043 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:44:46.673233 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:44:46.678333 13883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:44:46.678305 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:46.678437 13875 server_base.cc:1034] running on GCE node
W20260806 03:44:46.678324 13880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:44:46.678841 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:44:46.679086 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:44:46.680245 13875 hybrid_clock.cc:648] HybridClock initialized: now 1785987886680246 us; error 41 us; skew 500 ppm
Aug 06 03:44:46 dist-test-slave-10pc krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987886, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:44:46.682853 13875 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:44:46.683310 13875 webserver.cc:460] Webserver started at http://127.30.24.3:43575/ using document root <none> and password file <none>
I20260806 03:44:46.683538 13875 fs_manager.cc:359] Metadata directory not provided
I20260806 03:44:46.683606 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:44:46.685026 13875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:44:46.685586 13890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.685760 13875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.685861 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "7103a0b492924f59932fb88796259587"
format_stamp: "Formatted at 2026-08-06 03:44:45 on dist-test-slave-10pc"
server_key: "513078dd71b807ada4f8bc698497c941"
server_key_iv: "dc0305a8292cb89bd5113ac686b9558a"
server_key_version: "encryptionkey@0"
I20260806 03:44:46.686220 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:44:46.699607 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:44:46.701678 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:44:46.701853 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:44:46.702102 13875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:44:46.702419 13875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:44:46.702471 13875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.702507 13875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:44:46.702534 13875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:44:46.708480 13875 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37579
I20260806 03:44:46.708550 14008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37579 every 8 connection(s)
I20260806 03:44:46.708913 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:44:46.709992 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.709620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:48927)
0806 03:44:46.709772 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.709850 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
W20260806 03:44:46.710424 14009 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:34817 (0 consecutive failures): Network error: Failed to ping master at 127.30.24.62:34817: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
I20260806 03:44:46.710690 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.710606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:43743)
0806 03:44:46.710646 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.710672 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260806 03:44:46.710951 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:46.710883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:39377)
0806 03:44:46.710918 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:46.710939 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 03:44:46.711066 14009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:44:46.711377 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 13875
I20260806 03:44:47.516137 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.515824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:53223)
0806 03:44:47.515969 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:47.516054 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260806 03:44:47.584324 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:42489)
0806 03:44:47.584226 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:47.584264 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260806 03:44:47.711632 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.711417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:44863)
0806 03:44:47.711545 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:47.711579 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260806 03:44:47.722657 14016 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:47.722808 14016 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:47.723135 14016 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
Aug 06 03:44:47 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987886, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.2@KRBTEST.COM
Aug 06 03:44:47 dist-test-slave-10pc krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987886, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.3@KRBTEST.COM
I20260806 03:44:47.735787 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.723490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:43541 (local address 127.30.24.3:37579)
0806 03:44:47.723657 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:47.723664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:47.723692 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:44:47.724248 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:47.724251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:47.724310 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:47.724493 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:47.725458 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.726863 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.728361 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.729008 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.729132 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:47.732223 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:47.732227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:47.732227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:47.732238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:47.733914 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:47.734359 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:47.734361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:47.734362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:47.734471 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:47.734753 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:47.734753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:47.734754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:47.735069 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:47.735182 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:47.735525 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 03:44:47.735670 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260806 03:44:47.735785 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.723374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:55077 (local address 127.30.24.2:45081)
0806 03:44:47.723586 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 03:44:47.723600 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 03:44:47.723634 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:44:47.724249 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:44:47.724251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:44:47.724310 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:44:47.724485 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:44:47.725457 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.726855 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.728361 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.729008 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.729151 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:47.731782 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:44:47.731787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:44:47.731787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:44:47.731799 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:44:47.733733 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:47.734305 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:47.734308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:47.734309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:47.734436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:44:47.734753 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:44:47.734754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:44:47.734754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:44:47.735069 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:44:47.735157 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 03:44:47.735525 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 03:44:47.735670 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260806 03:44:47.735803 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.723265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:45081 (local address 127.30.24.1:55077)
0806 03:44:47.723420 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:47.723433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:47.723494 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:47.724545 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:47.724548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:47.725062 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:47.725333 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:47.725341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.726969 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.726970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:47.728266 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:47.728269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.729118 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.729119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:47.729254 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:47.729886 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:47.729906 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:47.731666 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:47.733816 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:47.733823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:47.733826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:47.734198 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:47.734519 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:47.734520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:47.734520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:47.734678 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:47.735166 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:47.735173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:47.735336 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 03:44:47.735453 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 03:44:47.735665 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":115}
I20260806 03:44:47.736135 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:47.723321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37579 (local address 127.30.24.1:43541)
0806 03:44:47.723439 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:47.723450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:44:47.723506 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:44:47.724545 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:44:47.724548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:44:47.725063 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:44:47.725333 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:47.725341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.726972 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.726972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:47.728266 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:44:47.728269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:44:47.729116 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:44:47.729117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:44:47.729254 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:44:47.729932 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:44:47.729938 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:44:47.732098 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:44:47.733991 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:47.733992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:47.733992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:47.734266 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:47.734536 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:44:47.734537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:44:47.734538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:44:47.734679 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:44:47.735181 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:44:47.735182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:44:47.735336 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 03:44:47.735453 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 03:44:47.735724 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":167,"thread_start_us":53,"threads_started":1}
W20260806 03:44:47.737071 13621 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
W20260806 03:44:47.737190 13623 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:47.737263 13623 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e; no voters: 7103a0b492924f59932fb88796259587, c45122be1aec43ecaa5491ce834263d7
I20260806 03:44:47.737385 14016 raft_consensus.cc:2726] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:44:48.516952 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:48.516699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:43019)
0806 03:44:48.516856 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:48.516900 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260806 03:44:48.585166 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:48.584854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:33223)
0806 03:44:48.584992 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:48.585101 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260806 03:44:48.712253 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:48.712073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:51725)
0806 03:44:48.712185 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:48.712215 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260806 03:44:49.280158 14023 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:49.280268 14023 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:49.280501 14023 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
W20260806 03:44:49.281049 13623 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
W20260806 03:44:49.281131 13621 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:49.281226 13621 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e; no voters: 7103a0b492924f59932fb88796259587, c45122be1aec43ecaa5491ce834263d7
I20260806 03:44:49.281313 14023 raft_consensus.cc:2726] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:44:49.517674 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:49.517450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:52093)
0806 03:44:49.517581 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:49.517624 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260806 03:44:49.585804 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:49.585605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:32779)
0806 03:44:49.585728 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:49.585764 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260806 03:44:49.712898 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:49.712688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:50487)
0806 03:44:49.712821 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:49.712859 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260806 03:44:50.518409 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:50.518175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:52959)
0806 03:44:50.518316 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:50.518357 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260806 03:44:50.586499 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:50.586287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:37751)
0806 03:44:50.586418 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:50.586457 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260806 03:44:50.713564 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:50.713371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:36077)
0806 03:44:50.713488 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:50.713524 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260806 03:44:51.519138 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:51.518929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:56787)
0806 03:44:51.519055 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:51.519091 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260806 03:44:51.587113 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:51.586920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:55091)
0806 03:44:51.587038 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:51.587067 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260806 03:44:51.714253 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:51.714028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:41837)
0806 03:44:51.714174 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:51.714210 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260806 03:44:52.519924 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:52.519656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:38133)
0806 03:44:52.519841 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:52.519875 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260806 03:44:52.528177 14034 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:52.528273 14034 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:52.528496 14034 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
W20260806 03:44:52.528908 13621 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
W20260806 03:44:52.529002 13623 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:52.529078 13623 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e; no voters: 7103a0b492924f59932fb88796259587, c45122be1aec43ecaa5491ce834263d7
I20260806 03:44:52.529158 14034 raft_consensus.cc:2726] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:44:52.587955 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:52.587709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:39751)
0806 03:44:52.587868 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:52.587910 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260806 03:44:52.714947 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:52.714719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:50927)
0806 03:44:52.714862 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:52.714902 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260806 03:44:53.520758 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:53.520449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:50623)
0806 03:44:53.520653 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:53.520710 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260806 03:44:53.588716 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:53.588440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:52043)
0806 03:44:53.588625 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:53.588665 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260806 03:44:53.715597 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:53.715389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:45189)
0806 03:44:53.715513 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:53.715551 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260806 03:44:54.128180 14040 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:54.128283 14040 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:54.128532 14040 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
W20260806 03:44:54.128959 13623 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
W20260806 03:44:54.129042 13621 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:54.129154 13621 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e; no voters: 7103a0b492924f59932fb88796259587, c45122be1aec43ecaa5491ce834263d7
I20260806 03:44:54.129251 14040 raft_consensus.cc:2726] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:44:54.521476 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:54.521239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:57807)
0806 03:44:54.521383 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:54.521426 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260806 03:44:54.589421 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:54.589162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:53439)
0806 03:44:54.589315 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:54.589356 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260806 03:44:54.716385 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:54.716148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:37627)
0806 03:44:54.716298 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:54.716339 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260806 03:44:55.522248 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:55.522022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:57317)
0806 03:44:55.522161 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:55.522197 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260806 03:44:55.590049 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:55.589843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:59513)
0806 03:44:55.589968 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:55.590004 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260806 03:44:55.717109 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:55.716880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:59637)
0806 03:44:55.717030 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:55.717067 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260806 03:44:56.522949 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:56.522706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:41165)
0806 03:44:56.522858 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:56.522897 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260806 03:44:56.590792 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:56.590545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:33059)
0806 03:44:56.590700 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:56.590743 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260806 03:44:56.632194 14049 raft_consensus.cc:482] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:56.632294 14049 raft_consensus.cc:504] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45122be1aec43ecaa5491ce834263d7" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 45081 } } peers { permanent_uuid: "7103a0b492924f59932fb88796259587" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37579 } } peers { permanent_uuid: "a69d5f7783254b9c9ab30dd72336db9e" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 42217 } }
I20260806 03:44:56.632548 14049 leader_election.cc:288] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081), 7103a0b492924f59932fb88796259587 (127.30.24.3:37579)
W20260806 03:44:56.633002 13623 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c45122be1aec43ecaa5491ce834263d7 (127.30.24.2:45081): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
W20260806 03:44:56.633103 13621 leader_election.cc:341] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7103a0b492924f59932fb88796259587 (127.30.24.3:37579): Not found: Tablet not found: d0391f35acc948f0ab4c6eba205d48e5
I20260806 03:44:56.633155 13621 leader_election.cc:302] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a69d5f7783254b9c9ab30dd72336db9e; no voters: 7103a0b492924f59932fb88796259587, c45122be1aec43ecaa5491ce834263d7
I20260806 03:44:56.633249 14049 raft_consensus.cc:2726] T d0391f35acc948f0ab4c6eba205d48e5 P a69d5f7783254b9c9ab30dd72336db9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:44:56.717852 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:56.717619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:44539)
0806 03:44:56.717765 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:56.717804 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260806 03:44:57.523718 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:57.523504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:35017)
0806 03:44:57.523629 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:57.523668 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260806 03:44:57.591535 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:57.591298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:53627)
0806 03:44:57.591434 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:57.591478 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260806 03:44:57.718643 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:57.718433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:33365)
0806 03:44:57.718566 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:57.718603 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260806 03:44:58.524554 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:58.524266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:37327)
0806 03:44:58.524416 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:58.524500 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260806 03:44:58.592257 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:58.592031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:35993)
0806 03:44:58.592173 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:58.592212 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260806 03:44:58.719434 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:58.719220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:39307)
0806 03:44:58.719346 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:58.719388 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260806 03:44:59.525272 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:59.525051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:41081)
0806 03:44:59.525183 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:59.525223 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260806 03:44:59.592979 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:59.592736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:51709)
0806 03:44:59.592888 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:59.592931 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260806 03:44:59.720227 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:44:59.719962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:51397)
0806 03:44:59.720144 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 03:44:59.720179 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
I20260806 03:45:00.526060 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:00.525787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:46729)
0806 03:45:00.525934 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:00.525978 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260806 03:45:00.593783 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:00.593511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:58645)
0806 03:45:00.593664 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:00.593708 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260806 03:45:00.721032 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:00.720805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:58425)
0806 03:45:00.720943 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:00.720985 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260806 03:45:01.526844 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:01.526632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.1:35997)
0806 03:45:01.526762 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:01.526800 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260806 03:45:01.594702 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:01.594360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.2:50997)
0806 03:45:01.594493 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:01.594532 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260806 03:45:01.720712 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 13574
I20260806 03:45:01.722306 14066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:01.721710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:34817 (local address 127.30.24.3:46311)
0806 03:45:01.722231 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:01.722265 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":54,"threads_started":1}
I20260806 03:45:01.727687 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 13740
I20260806 03:45:01.733367 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 13875
2026-08-06T03:45:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:45:01 dist-test-slave-10pc krb5kdc[14069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:45:01 dist-test-slave-10pc krb5kdc[14069](info): set up 2 sockets
Aug 06 03:45:01 dist-test-slave-10pc krb5kdc[14069](info): commencing operation
krb5kdc: starting...
W20260806 03:45:02.790084 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:45:02 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:45:02.915094 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37183
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:02.999466 14082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:02.999701 14082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:02.999724 14082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:03.000990 14082 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 03:45:03.001037 14082 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 03:45:03.001051 14082 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 03:45:03.001063 14082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:45:03.001118 14082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:03.001138 14082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:03.001160 14082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:45:03.001184 14082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:45:03.002696 14082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:03.002964 14082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:03.003216 14082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:03.008237 14089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:03.008280 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.008285 14082 server_base.cc:1034] running on GCE node
W20260806 03:45:03.008215 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.008657 14082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:03.009063 14082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:03.010278 14082 hybrid_clock.cc:648] HybridClock initialized: now 1785987903010251 us; error 41 us; skew 500 ppm
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:03.013375 14082 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:45:03.013877 14082 webserver.cc:460] Webserver started at http://127.30.24.62:40301/ using document root <none> and password file <none>
I20260806 03:45:03.014106 14082 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:03.014175 14082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:03.014310 14082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:45:03.015393 14082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "f42699a521e54df3b62a9b7ff9bde11e"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "e511ab112f3b881bffe22eac11863aaa"
server_key_iv: "181d2a202645048b1f7edef060265869"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.015774 14082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "f42699a521e54df3b62a9b7ff9bde11e"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "e511ab112f3b881bffe22eac11863aaa"
server_key_iv: "181d2a202645048b1f7edef060265869"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.017218 14082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:45:03.017992 14098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.018194 14082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:45:03.018298 14082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "f42699a521e54df3b62a9b7ff9bde11e"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "e511ab112f3b881bffe22eac11863aaa"
server_key_iv: "181d2a202645048b1f7edef060265869"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.018371 14082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:03.028362 14082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:03.030753 14082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:03.030866 14082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:03.034922 14082 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:37183
I20260806 03:45:03.034956 14150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:37183 every 8 connection(s)
I20260806 03:45:03.035338 14082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:45:03.036186 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:03.038496 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap starting.
I20260806 03:45:03.039067 14151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:03.039321 14151 log.cc:824] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:03.039950 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: No bootstrap required, opened a new log
I20260806 03:45:03.040642 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14082
I20260806 03:45:03.040761 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:45:03.040946 30816 external_mini_cluster.cc:1428] Setting key cf3b813b0511a231d5c804863bac1080
I20260806 03:45:03.042007 14151 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:03.042201 14151 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:03.042284 14151 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42699a521e54df3b62a9b7ff9bde11e, State: Initialized, Role: FOLLOWER
I20260806 03:45:03.042431 14151 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:03.042523 14151 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:45:03.042569 14151 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:45:03.042626 14151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:03.043294 14151 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:03.043385 14151 leader_election.cc:302] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42699a521e54df3b62a9b7ff9bde11e; no voters:
I20260806 03:45:03.043607 14151 leader_election.cc:288] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:45:03.043653 14156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:45:03.043814 14156 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 LEADER]: Becoming Leader. State: Replica: f42699a521e54df3b62a9b7ff9bde11e, State: Running, Role: LEADER
I20260806 03:45:03.043879 14151 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:45:03.043951 14156 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:03.044279 14156 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42699a521e54df3b62a9b7ff9bde11e. Latest consensus state: current_term: 1 leader_uuid: "f42699a521e54df3b62a9b7ff9bde11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } } }
I20260806 03:45:03.044382 14156 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: This master's current role is: LEADER
I20260806 03:45:03.044291 14157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42699a521e54df3b62a9b7ff9bde11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } } }
I20260806 03:45:03.044570 14157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: This master's current role is: LEADER
I20260806 03:45:03.044646 14163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:45:03.045198 14163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:45:03.046947 14163 catalog_manager.cc:1347] Generated new cluster ID: 11e4fa43d31b4ee9b9faf670c9b06d00
I20260806 03:45:03.047000 14163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:03.052392 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.041478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.30.24.62:37183)
0806 03:45:03.041677 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.041684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.041709 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:45:03.041985 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.041993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.042037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.042149 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:03.043023 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.044652 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.045752 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.046352 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.046458 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.049108 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:03.049112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:03.049112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:03.049132 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:03.050802 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.051180 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.051184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.051184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.051292 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.051516 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.051517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.051517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.051754 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:03.051911 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.052025 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.052219 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260806 03:45:03.079429 14163 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:45:03.080219 14163 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:45:03.084637 14163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Generated new TSK 0
I20260806 03:45:03.084872 14163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:45:03.119740 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37183
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:03.206167 14178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:03.206386 14178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:03.206410 14178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:03.207522 14178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:03.207562 14178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:03.207595 14178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:45:03.209082 14178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:03.209277 14178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:03.209461 14178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:03.214592 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:03.214605 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.214627 14178 server_base.cc:1034] running on GCE node
W20260806 03:45:03.214605 14186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.215093 14178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:03.215318 14178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:03.216523 14178 hybrid_clock.cc:648] HybridClock initialized: now 1785987903216484 us; error 37 us; skew 500 ppm
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:03.219219 14178 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:45:03.219697 14178 webserver.cc:460] Webserver started at http://127.30.24.1:35757/ using document root <none> and password file <none>
I20260806 03:45:03.219928 14178 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:03.219975 14178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:03.220124 14178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:45:03.221355 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "da348e5b0f0744aa8409df3cc5d98e55"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "2066dccf44fc19d4096bcb61c1d5ec6f"
server_key_iv: "56edfe836314b471c3264f483f60b6b9"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.221740 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "da348e5b0f0744aa8409df3cc5d98e55"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "2066dccf44fc19d4096bcb61c1d5ec6f"
server_key_iv: "56edfe836314b471c3264f483f60b6b9"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.223163 14178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:45:03.223943 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.224135 14178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:45:03.224231 14178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "da348e5b0f0744aa8409df3cc5d98e55"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "2066dccf44fc19d4096bcb61c1d5ec6f"
server_key_iv: "56edfe836314b471c3264f483f60b6b9"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.224289 14178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:03.236663 14178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:03.238942 14178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:03.239065 14178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:03.239267 14178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:03.239617 14178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:03.239655 14178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.239677 14178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:03.239691 14178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.246037 14178 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:39903
I20260806 03:45:03.246124 14307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:39903 every 8 connection(s)
I20260806 03:45:03.246507 14178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:03.255776 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14178
I20260806 03:45:03.255898 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:45:03.256073 30816 external_mini_cluster.cc:1428] Setting key 0a4cf6e56ed633fe2341e14bebffc645
I20260806 03:45:03.256595 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.247397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:57661 (local address 127.30.24.62:37183)
0806 03:45:03.247485 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.247487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.247610 (+ 123us) server_negotiation.cc:407] Connection header received
0806 03:45:03.248202 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.248204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.248219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.248277 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:03.249326 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.249830 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.250853 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.251043 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.251096 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.253493 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:03.253494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:03.253494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:03.253501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:03.254881 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.255265 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.255266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.255266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.255331 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.255636 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.255637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.255638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.255796 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:03.255882 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.256406 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.256523 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 03:45:03.256814 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.247318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:57661)
0806 03:45:03.247488 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.247530 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.248062 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.248336 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.248340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.248994 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:03.249222 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.249227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.249922 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.249923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.250774 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.250776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.251148 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.251149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.251262 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.251855 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:03.251864 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:03.253389 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:03.254960 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:03.254966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:03.254969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:03.255190 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:03.255418 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:03.255419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:03.255419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:03.255550 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:03.256033 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:03.256041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:03.256216 (+ 175us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.256351 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.256587 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260806 03:45:03.257383 14308 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37183
I20260806 03:45:03.257491 14308 heartbeater.cc:464] Registering TS with master...
I20260806 03:45:03.257704 14308 heartbeater.cc:511] Master 127.30.24.62:37183 requested a full tablet report, sending...
I20260806 03:45:03.258255 14115 ts_manager.cc:194] Registered new tserver with Master: da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903)
I20260806 03:45:03.259099 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:57661
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:45:03.284291 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37183
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:03.369812 14313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:03.370041 14313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:03.370067 14313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:03.371253 14313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:03.371299 14313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:03.371326 14313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:45:03.372814 14313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:03.373075 14313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:03.373325 14313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:03.378463 14319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.378543 14313 server_base.cc:1034] running on GCE node
W20260806 03:45:03.378485 14318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:03.378476 14322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.378892 14313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:03.379155 14313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:03.380301 14313 hybrid_clock.cc:648] HybridClock initialized: now 1785987903380280 us; error 39 us; skew 500 ppm
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:03.382915 14313 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:45:03.383389 14313 webserver.cc:460] Webserver started at http://127.30.24.2:36421/ using document root <none> and password file <none>
I20260806 03:45:03.383631 14313 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:03.383679 14313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:03.383816 14313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:45:03.384974 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "b220811780e9474a844771ed26dd20ed"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "97a00392fb062ede72529d2047da0ea8"
server_key_iv: "d08200b0daa4ea8334b872be92af9bcf"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.385357 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "b220811780e9474a844771ed26dd20ed"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "97a00392fb062ede72529d2047da0ea8"
server_key_iv: "d08200b0daa4ea8334b872be92af9bcf"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.386696 14313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:45:03.387441 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.387668 14313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:03.387774 14313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "b220811780e9474a844771ed26dd20ed"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "97a00392fb062ede72529d2047da0ea8"
server_key_iv: "d08200b0daa4ea8334b872be92af9bcf"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.387853 14313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:03.402406 14313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:03.404615 14313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:03.404791 14313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:03.405037 14313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:03.405375 14313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:03.405411 14313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.405455 14313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:03.405483 14313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.411689 14313 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:43659
I20260806 03:45:03.411767 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:43659 every 8 connection(s)
I20260806 03:45:03.412104 14313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:03.420151 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14313
I20260806 03:45:03.420286 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:45:03.420545 30816 external_mini_cluster.cc:1428] Setting key bd8a29b8d12c04f45878b70a6df02482
I20260806 03:45:03.422259 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.412886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:34273 (local address 127.30.24.62:37183)
0806 03:45:03.412930 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.412932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.413207 (+ 275us) server_negotiation.cc:407] Connection header received
0806 03:45:03.413757 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.413758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.413772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.413829 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:03.414848 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.415364 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.416405 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.416694 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.416777 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.418956 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:03.418958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:03.418958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:03.418965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:03.420589 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.421135 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.421136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.421136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.421208 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.421471 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.421472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.421473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.421653 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:03.421738 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.422067 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.422166 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:45:03.422457 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.412853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:34273)
0806 03:45:03.413040 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.413120 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.413645 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.413903 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.413907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.414542 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:03.414749 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.414753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.415480 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.415481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.416298 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.416300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.416785 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.416787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.416920 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.417425 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:03.417434 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:03.418857 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:03.420711 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:03.420727 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:03.420729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:03.421047 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:03.421278 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:03.421279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:03.421280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:03.421402 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:03.421729 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:03.421736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:03.421919 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.422046 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.422222 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260806 03:45:03.422991 14446 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37183
I20260806 03:45:03.423127 14446 heartbeater.cc:464] Registering TS with master...
I20260806 03:45:03.423321 14446 heartbeater.cc:511] Master 127.30.24.62:37183 requested a full tablet report, sending...
I20260806 03:45:03.423710 14115 ts_manager.cc:194] Registered new tserver with Master: b220811780e9474a844771ed26dd20ed (127.30.24.2:43659)
I20260806 03:45:03.424250 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:34273
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:45:03.448148 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37183
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:03.532565 14451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:03.532774 14451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:03.532799 14451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:03.533958 14451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:03.534001 14451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:03.534099 14451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:45:03.535589 14451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:03.535832 14451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:03.536074 14451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:03.541275 14457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:03.541283 14456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:03.541280 14459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:03.541600 14451 server_base.cc:1034] running on GCE node
I20260806 03:45:03.541848 14451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:03.542068 14451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:03.543192 14451 hybrid_clock.cc:648] HybridClock initialized: now 1785987903543178 us; error 28 us; skew 500 ppm
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:03.546036 14451 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:45:03.546473 14451 webserver.cc:460] Webserver started at http://127.30.24.3:38385/ using document root <none> and password file <none>
I20260806 03:45:03.546691 14451 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:03.546734 14451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:03.546823 14451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:45:03.547881 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "c3061a896ab64e04ae9479760844105f"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "9412329e4722ecd0aeafea16e85e19f5"
server_key_iv: "8a0e752eec27d55e9b5e51a0d9d51bf4"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.548200 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "c3061a896ab64e04ae9479760844105f"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "9412329e4722ecd0aeafea16e85e19f5"
server_key_iv: "8a0e752eec27d55e9b5e51a0d9d51bf4"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.549578 14451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:45:03.550321 14466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.550493 14451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:45:03.550606 14451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "c3061a896ab64e04ae9479760844105f"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "9412329e4722ecd0aeafea16e85e19f5"
server_key_iv: "8a0e752eec27d55e9b5e51a0d9d51bf4"
server_key_version: "encryptionkey@0"
I20260806 03:45:03.550690 14451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:03.575639 14451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:03.578150 14451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:03.578329 14451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:03.578599 14451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:03.578938 14451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:03.578994 14451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.579034 14451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:03.579077 14451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:03.585245 14451 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37389
I20260806 03:45:03.585314 14582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37389 every 8 connection(s)
I20260806 03:45:03.585640 14451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:45:03 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:03.594005 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14451
I20260806 03:45:03.594122 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:45:03.594328 30816 external_mini_cluster.cc:1428] Setting key be3818b46d08c6fa8485c03cc27433df
I20260806 03:45:03.595548 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.586397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:36249 (local address 127.30.24.62:37183)
0806 03:45:03.586466 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.586468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.586702 (+ 234us) server_negotiation.cc:407] Connection header received
0806 03:45:03.587179 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.587181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.587192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.587251 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:03.588243 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.588827 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.589871 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.590115 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.590184 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.592579 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:03.592580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:03.592581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:03.592588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:03.593859 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.594395 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.594396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.594397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.594471 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.594822 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.594823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.594823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.594969 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:03.595040 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.595402 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.595483 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:45:03.595710 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.586399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:36249)
0806 03:45:03.586558 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.586595 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.587080 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.587334 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.587339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.587911 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:03.588115 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.588119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.588919 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.588920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.589763 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.589766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.590200 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.590201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.590310 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.590920 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:03.590931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:03.592439 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:03.593942 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:03.593951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:03.593953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:03.594301 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:03.594556 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:03.594557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:03.594558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:03.594723 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:03.595049 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:03.595056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:03.595247 (+ 191us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.595364 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.595501 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260806 03:45:03.596145 14583 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37183
I20260806 03:45:03.596238 14583 heartbeater.cc:464] Registering TS with master...
I20260806 03:45:03.596390 14583 heartbeater.cc:511] Master 127.30.24.62:37183 requested a full tablet report, sending...
I20260806 03:45:03.596771 14115 ts_manager.cc:194] Registered new tserver with Master: c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
I20260806 03:45:03.597101 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:36249
I20260806 03:45:03.606208 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:45:03.613868 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.607589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.30.24.62:37183)
0806 03:45:03.607671 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.607674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.607790 (+ 116us) server_negotiation.cc:407] Connection header received
0806 03:45:03.607880 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.607881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.607891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.607942 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:03.608557 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.609060 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.609788 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.609997 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.610072 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.611239 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:03.611240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:03.611240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:03.611248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:03.612696 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.613046 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.613047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.613048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.613108 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:03.613324 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:03.613325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:03.613325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:03.613475 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:03.613590 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.613688 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.613766 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 03:45:03.615456 14115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:45:03.617152 14115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:45:03.623911 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.619589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:43659 (local address 127.0.0.1:36492)
0806 03:45:03.620017 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.620026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.620075 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.620352 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.620353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.620358 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:03.620403 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.620404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.621677 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.621678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.622725 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.622728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.623611 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.623612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.623672 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.623684 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.623742 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.623815 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260806 03:45:03.623909 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.619734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.30.24.2:43659)
0806 03:45:03.619924 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.619928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.620130 (+ 202us) server_negotiation.cc:407] Connection header received
0806 03:45:03.620179 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.620181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.620198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.620271 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:03.620505 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.621562 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.622820 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.623524 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.623605 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.623675 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.623776 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.623818 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260806 03:45:03.624012 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.619388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:39903 (local address 127.0.0.1:56002)
0806 03:45:03.619924 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.619936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.619992 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.620352 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.620353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.620355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:03.620408 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.620410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.621506 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.621506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.622918 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.622919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.623802 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.623803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.623854 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.623857 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.623906 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.623945 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":63,"spinlock_wait_cycles":768,"thread_start_us":96,"threads_started":1}
I20260806 03:45:03.624190 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.619769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.30.24.1:39903)
0806 03:45:03.619964 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.619968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.619979 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:45:03.620098 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.620101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.620119 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.620243 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:03.620532 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.621380 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.623012 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.623715 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.623790 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.623845 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.624094 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.624130 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20260806 03:45:03.624804 14377 tablet_service.cc:1467] Processing CreateTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 (DEFAULT_TABLE table=test-table [id=5a4e3c95ff1c4f91a8665802dae167d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:45:03.624917 14241 tablet_service.cc:1467] Processing CreateTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 (DEFAULT_TABLE table=test-table [id=5a4e3c95ff1c4f91a8665802dae167d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:45:03.625229 14377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e1e937d76846bbbd30b3327c6018f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:03.625293 14241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e1e937d76846bbbd30b3327c6018f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:03.625628 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.619388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37389 (local address 127.0.0.1:46618)
0806 03:45:03.619557 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.619601 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.619671 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.619980 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.619983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.619985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:03.620072 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.620074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.621502 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.621503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.622916 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.622919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.625215 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.625216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.625335 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.625339 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.625481 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.625525 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":57,"thread_start_us":56,"threads_started":1}
I20260806 03:45:03.625656 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.619498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.30.24.3:37389)
0806 03:45:03.619727 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.619732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.619742 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:45:03.619778 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.619781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.619795 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.619892 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:03.620156 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.621381 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.623882 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.625044 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.625143 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.625201 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.625519 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.625563 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260806 03:45:03.626647 14516 tablet_service.cc:1467] Processing CreateTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 (DEFAULT_TABLE table=test-table [id=5a4e3c95ff1c4f91a8665802dae167d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:45:03.626978 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e1e937d76846bbbd30b3327c6018f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:03.628440 14600 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Bootstrap starting.
I20260806 03:45:03.629050 14601 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap starting.
I20260806 03:45:03.629297 14600 tablet_bootstrap.cc:654] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:03.629827 14600 log.cc:824] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:03.629901 14601 tablet_bootstrap.cc:654] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:03.630486 14601 log.cc:824] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:03.630671 14603 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Bootstrap starting.
I20260806 03:45:03.630745 14600 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: No bootstrap required, opened a new log
I20260806 03:45:03.630817 14600 ts_tablet_manager.cc:1397] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:45:03.631163 14601 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: No bootstrap required, opened a new log
I20260806 03:45:03.631246 14601 ts_tablet_manager.cc:1397] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:45:03.631352 14603 tablet_bootstrap.cc:654] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:03.631935 14603 log.cc:824] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:03.632743 14603 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: No bootstrap required, opened a new log
I20260806 03:45:03.632817 14603 ts_tablet_manager.cc:1397] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:45:03.633024 14600 raft_consensus.cc:348] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.633208 14600 raft_consensus.cc:374] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:03.633242 14600 raft_consensus.cc:729] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b220811780e9474a844771ed26dd20ed, State: Initialized, Role: FOLLOWER
I20260806 03:45:03.633234 14601 raft_consensus.cc:348] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.633386 14600 consensus_queue.cc:260] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.633424 14601 raft_consensus.cc:374] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:03.633466 14601 raft_consensus.cc:729] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da348e5b0f0744aa8409df3cc5d98e55, State: Initialized, Role: FOLLOWER
I20260806 03:45:03.633615 14601 consensus_queue.cc:260] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.633663 14600 ts_tablet_manager.cc:1428] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:45:03.633805 14446 heartbeater.cc:503] Master 127.30.24.62:37183 was elected leader, sending a full tablet report...
I20260806 03:45:03.633893 14601 ts_tablet_manager.cc:1428] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:45:03.634011 14308 heartbeater.cc:503] Master 127.30.24.62:37183 was elected leader, sending a full tablet report...
I20260806 03:45:03.634702 14603 raft_consensus.cc:348] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.634862 14603 raft_consensus.cc:374] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:03.634909 14603 raft_consensus.cc:729] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3061a896ab64e04ae9479760844105f, State: Initialized, Role: FOLLOWER
I20260806 03:45:03.635023 14603 consensus_queue.cc:260] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.635324 14603 ts_tablet_manager.cc:1428] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 03:45:03.635349 14583 heartbeater.cc:503] Master 127.30.24.62:37183 was elected leader, sending a full tablet report...
W20260806 03:45:03.662703 14447 tablet.cc:2369] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:45:03.747296 14309 tablet.cc:2369] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:45:03.836246 14584 tablet.cc:2369] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:45:03.947378 14606 raft_consensus.cc:482] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:03.947515 14606 raft_consensus.cc:504] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.947854 14606 leader_election.cc:288] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
I20260806 03:45:03.951009 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.948250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:49969 (local address 127.30.24.3:37389)
0806 03:45:03.948313 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.948316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.948330 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:45:03.948364 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.948365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.948378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.948440 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:03.948667 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.949171 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.950230 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.950620 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.950713 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.950724 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.950893 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.950933 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 03:45:03.950996 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.948173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:53999 (local address 127.30.24.1:39903)
0806 03:45:03.948225 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.948227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.948348 (+ 121us) server_negotiation.cc:407] Connection header received
0806 03:45:03.948401 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.948402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.948432 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.948522 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:03.948703 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.949169 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.950270 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.950627 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.950709 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.950721 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.950875 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.950917 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:45:03.951023 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.948100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37389 (local address 127.30.24.2:49969)
0806 03:45:03.948226 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.948238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.948299 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.948485 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.948487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.948490 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:03.948595 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.948597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.949258 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.949259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.950138 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.950140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.950717 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.950718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.950799 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.950805 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.950879 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.950921 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":26,"thread_start_us":63,"threads_started":1}
I20260806 03:45:03.951026 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.948086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:39903 (local address 127.30.24.2:53999)
0806 03:45:03.948221 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:03.948235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:03.948340 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:03.948532 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:03.948532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:03.948534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:03.948603 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.948605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.949262 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.949263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.950147 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:03.950149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.950725 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.950726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:03.950799 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.950802 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:45:03.950843 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 03:45:03.950900 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":26,"thread_start_us":66,"threads_started":1}
I20260806 03:45:03.951467 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e1e937d76846bbbd30b3327c6018f3" candidate_uuid: "b220811780e9474a844771ed26dd20ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da348e5b0f0744aa8409df3cc5d98e55" is_pre_election: true
I20260806 03:45:03.951576 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e1e937d76846bbbd30b3327c6018f3" candidate_uuid: "b220811780e9474a844771ed26dd20ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3061a896ab64e04ae9479760844105f" is_pre_election: true
I20260806 03:45:03.951661 14261 raft_consensus.cc:2454] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b220811780e9474a844771ed26dd20ed in term 0.
I20260806 03:45:03.951771 14536 raft_consensus.cc:2454] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b220811780e9474a844771ed26dd20ed in term 0.
I20260806 03:45:03.951917 14331 leader_election.cc:302] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b220811780e9474a844771ed26dd20ed, da348e5b0f0744aa8409df3cc5d98e55; no voters:
I20260806 03:45:03.952184 14606 raft_consensus.cc:2781] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:45:03.952229 14606 raft_consensus.cc:482] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:45:03.952244 14606 raft_consensus.cc:3037] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:03.953065 14606 raft_consensus.cc:504] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.953225 14606 leader_election.cc:288] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [CANDIDATE]: Term 1 election: Requested vote from peers da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
I20260806 03:45:03.953403 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e1e937d76846bbbd30b3327c6018f3" candidate_uuid: "b220811780e9474a844771ed26dd20ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da348e5b0f0744aa8409df3cc5d98e55"
I20260806 03:45:03.953410 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e1e937d76846bbbd30b3327c6018f3" candidate_uuid: "b220811780e9474a844771ed26dd20ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3061a896ab64e04ae9479760844105f"
I20260806 03:45:03.953486 14261 raft_consensus.cc:3037] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:03.953486 14536 raft_consensus.cc:3037] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:03.954397 14261 raft_consensus.cc:2454] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b220811780e9474a844771ed26dd20ed in term 1.
I20260806 03:45:03.954397 14536 raft_consensus.cc:2454] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b220811780e9474a844771ed26dd20ed in term 1.
I20260806 03:45:03.954571 14331 leader_election.cc:302] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b220811780e9474a844771ed26dd20ed, da348e5b0f0744aa8409df3cc5d98e55; no voters:
I20260806 03:45:03.954660 14606 raft_consensus.cc:2781] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:45:03.954829 14606 raft_consensus.cc:686] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 1 LEADER]: Becoming Leader. State: Replica: b220811780e9474a844771ed26dd20ed, State: Running, Role: LEADER
I20260806 03:45:03.954932 14606 consensus_queue.cc:237] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:03.955826 14114 catalog_manager.cc:5495] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed reported cstate change: term changed from 0 to 1, leader changed from <none> to b220811780e9474a844771ed26dd20ed (127.30.24.2). New cstate: current_term: 1 leader_uuid: "b220811780e9474a844771ed26dd20ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } health_report { overall_health: UNKNOWN } } }
I20260806 03:45:03.981398 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:03.978892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.30.24.2:43659)
0806 03:45:03.978963 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:03.978965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:03.978974 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:45:03.979015 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:03.979016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:03.979046 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:03.979134 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:45:03.979292 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.979766 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.980573 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:03.980785 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:03.980872 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:03.981046 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:45:03.981162 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:45:03.981250 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:03.981289 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 03:45:03.981310 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:45:03.983381 14261 raft_consensus.cc:1261] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Refusing update from remote peer b220811780e9474a844771ed26dd20ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:45:03.983414 14536 raft_consensus.cc:1261] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Refusing update from remote peer b220811780e9474a844771ed26dd20ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:45:03.983793 14606 consensus_queue.cc:1035] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:45:03.983964 14611 consensus_queue.cc:1035] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:45:03.986375 14614 mvcc.cc:205] Tried to move back new op lower bound from 7315406454713040896 to 7315406454600261632. Current Snapshot: MvccSnapshot[applied={T|T < 7315406454713040896}]
I20260806 03:45:03.986922 14615 mvcc.cc:205] Tried to move back new op lower bound from 7315406454713040896 to 7315406454600261632. Current Snapshot: MvccSnapshot[applied={T|T < 7315406454713040896}]
I20260806 03:45:03.989658 14114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:45:03.989781 14114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:45:03.990648 14114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Sending DeleteTablet for 3 replicas of tablet 59e1e937d76846bbbd30b3327c6018f3
I20260806 03:45:03.991057 14241 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:03 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:56002
I20260806 03:45:03.991117 14516 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:03 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:46618
I20260806 03:45:03.991393 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14178
I20260806 03:45:03.991436 14377 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:03 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:36492
I20260806 03:45:03.991430 14625 tablet_replica.cc:330] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: stopping tablet replica
I20260806 03:45:03.992381 14627 tablet_replica.cc:330] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: stopping tablet replica
I20260806 03:45:03.992549 14627 raft_consensus.cc:2229] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:45:03.992727 14627 raft_consensus.cc:2258] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:45:03.993361 14627 ts_tablet_manager.cc:1905] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:45:03.995375 14627 ts_tablet_manager.cc:1918] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:45:03.995445 14627 log.cc:1196] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/59e1e937d76846bbbd30b3327c6018f3
I20260806 03:45:03.995729 14627 ts_tablet_manager.cc:1939] T 59e1e937d76846bbbd30b3327c6018f3 P b220811780e9474a844771ed26dd20ed: Deleting consensus metadata
I20260806 03:45:03.995970 14626 tablet_replica.cc:330] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: stopping tablet replica
I20260806 03:45:03.996191 14100 catalog_manager.cc:4841] TS b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): tablet 59e1e937d76846bbbd30b3327c6018f3 (table test-table [id=5a4e3c95ff1c4f91a8665802dae167d4]) successfully deleted
I20260806 03:45:03.996222 14626 raft_consensus.cc:2229] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:45:03.996392 14626 raft_consensus.cc:2258] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:45:03.996870 14626 ts_tablet_manager.cc:1905] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:45:03.998237 14626 ts_tablet_manager.cc:1918] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:45:03.998338 14626 log.cc:1196] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/59e1e937d76846bbbd30b3327c6018f3
I20260806 03:45:03.998632 14626 ts_tablet_manager.cc:1939] T 59e1e937d76846bbbd30b3327c6018f3 P c3061a896ab64e04ae9479760844105f: Deleting consensus metadata
I20260806 03:45:03.999085 14102 catalog_manager.cc:4841] TS c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): tablet 59e1e937d76846bbbd30b3327c6018f3 (table test-table [id=5a4e3c95ff1c4f91a8665802dae167d4]) successfully deleted
W20260806 03:45:04.001492 14100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:39903 (error 108)
I20260806 03:45:04.002102 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:04.001952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:39903 (local address 127.0.0.1:56012)
0806 03:45:04.002016 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:04.002061 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:39903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 03:45:04.002169 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14313
W20260806 03:45:04.005435 14100 catalog_manager.cc:4584] TS da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59e1e937d76846bbbd30b3327c6018f3: Network error: Client connection negotiation failed: client connection to 127.30.24.1:39903: connect: Connection refused (error 111)
I20260806 03:45:04.008738 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14451
I20260806 03:45:04.017601 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14082
I20260806 03:45:24.024528 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37183
--webserver_interface=127.30.24.62
--webserver_port=40301
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:24.118906 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:24.119150 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:24.119175 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:24.120577 14629 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 03:45:24.120627 14629 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 03:45:24.120643 14629 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 03:45:24.120656 14629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:45:24.120723 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:24.120743 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:24.120770 14629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:45:24.120803 14629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:45:24.122486 14629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=40301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:24.122781 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:24.123076 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:24.133631 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.133694 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:24.133711 14629 server_base.cc:1034] running on GCE node
W20260806 03:45:24.133648 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:24.134189 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:24.134553 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:24.135769 14629 hybrid_clock.cc:648] HybridClock initialized: now 1785987924135743 us; error 41 us; skew 500 ppm
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:24.138625 14629 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:45:24.139058 14629 webserver.cc:460] Webserver started at http://127.30.24.62:40301/ using document root <none> and password file <none>
I20260806 03:45:24.139258 14629 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:24.139313 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:24.140851 14629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:24.141577 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.141784 14629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:24.141887 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "f42699a521e54df3b62a9b7ff9bde11e"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "e511ab112f3b881bffe22eac11863aaa"
server_key_iv: "181d2a202645048b1f7edef060265869"
server_key_version: "encryptionkey@0"
I20260806 03:45:24.142314 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:24.162587 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:24.164600 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:24.164767 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:24.168812 14629 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:37183
I20260806 03:45:24.168874 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:37183 every 8 connection(s)
I20260806 03:45:24.169281 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:45:24.170847 14699 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:45:24.171327 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14629
I20260806 03:45:24.171671 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:39903
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:37183
--webserver_port=35757
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:45:24.171685 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap starting.
I20260806 03:45:24.181010 14699 log.cc:824] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:24.182930 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:45:24.183264 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap complete.
I20260806 03:45:24.185822 14699 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:24.186168 14699 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42699a521e54df3b62a9b7ff9bde11e, State: Initialized, Role: FOLLOWER
I20260806 03:45:24.186388 14699 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:24.186491 14699 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:45:24.186551 14699 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:45:24.186600 14699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:45:24.187628 14699 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:24.187757 14699 leader_election.cc:302] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42699a521e54df3b62a9b7ff9bde11e; no voters:
I20260806 03:45:24.187988 14699 leader_election.cc:288] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:45:24.188024 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:45:24.188151 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 LEADER]: Becoming Leader. State: Replica: f42699a521e54df3b62a9b7ff9bde11e, State: Running, Role: LEADER
I20260806 03:45:24.188282 14699 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:45:24.188341 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:24.188773 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f42699a521e54df3b62a9b7ff9bde11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } } }
I20260806 03:45:24.188864 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: This master's current role is: LEADER
I20260806 03:45:24.189002 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42699a521e54df3b62a9b7ff9bde11e. Latest consensus state: current_term: 2 leader_uuid: "f42699a521e54df3b62a9b7ff9bde11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } } }
I20260806 03:45:24.189105 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: This master's current role is: LEADER
I20260806 03:45:24.190399 14719 catalog_manager.cc:1259] Loaded cluster ID: 11e4fa43d31b4ee9b9faf670c9b06d00
I20260806 03:45:24.190449 14719 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: loading cluster ID for follower catalog manager: success
I20260806 03:45:24.191648 14719 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: acquiring CA information for follower catalog manager: success
I20260806 03:45:24.192056 14719 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:45:24.192607 14708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:45:24.193075 14708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:45:24.193192 14708 catalog_manager.cc:1259] Loaded cluster ID: 11e4fa43d31b4ee9b9faf670c9b06d00
I20260806 03:45:24.193230 14708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:45:24.193416 14708 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:45:24.193504 14708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Loaded TSK: 0
I20260806 03:45:24.203037 14708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Generated new TSK 1
I20260806 03:45:24.203137 14708 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:45:24.262113 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:24.262363 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:24.262406 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:24.263592 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:24.263667 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:24.263713 14701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:45:24.265333 14701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:39903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=35757
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:24.265606 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:24.265856 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:24.270956 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.270967 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.271068 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:24.271243 14701 server_base.cc:1034] running on GCE node
I20260806 03:45:24.271512 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:24.271804 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:24.272972 14701 hybrid_clock.cc:648] HybridClock initialized: now 1785987924272940 us; error 48 us; skew 500 ppm
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:24.275771 14701 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:45:24.276319 14701 webserver.cc:460] Webserver started at http://127.30.24.1:35757/ using document root <none> and password file <none>
I20260806 03:45:24.276608 14701 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:24.276697 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:24.278128 14701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:24.278875 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.279157 14701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:24.279273 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "da348e5b0f0744aa8409df3cc5d98e55"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "2066dccf44fc19d4096bcb61c1d5ec6f"
server_key_iv: "56edfe836314b471c3264f483f60b6b9"
server_key_version: "encryptionkey@0"
I20260806 03:45:24.279673 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:24.303274 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:24.305612 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:24.305787 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:24.306074 14701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:24.306635 14743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 03:45:24.308259 14701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 03:45:24.308310 14701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 03:45:24.308346 14701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 03:45:24.309063 14701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 03:45:24.309110 14701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 03:45:24.309163 14743 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap starting.
I20260806 03:45:24.316932 14701 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:39903
I20260806 03:45:24.317441 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:45:24.318078 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14701
I20260806 03:45:24.318389 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:43659
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:37183
--webserver_port=36421
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:45:24.321524 14743 log.cc:824] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:24.321882 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:39903 every 8 connection(s)
I20260806 03:45:24.323905 14743 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:45:24.324324 14743 tablet_bootstrap.cc:492] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap complete.
I20260806 03:45:24.325632 14743 ts_tablet_manager.cc:1397] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20260806 03:45:24.327050 14743 raft_consensus.cc:348] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:24.327462 14743 raft_consensus.cc:729] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da348e5b0f0744aa8409df3cc5d98e55, State: Initialized, Role: FOLLOWER
I20260806 03:45:24.327692 14743 consensus_queue.cc:260] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } } peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } }
I20260806 03:45:24.328038 14743 ts_tablet_manager.cc:1428] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:24.335182 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.320546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:53105)
0806 03:45:24.323739 (+ 3193us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.323783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.324497 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.325156 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.325160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.325566 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:24.325874 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.325881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.327489 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.327490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.328665 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.328668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.329364 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.329365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.329498 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.330096 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:24.330105 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:24.331498 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:24.333486 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.333494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.333496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.333769 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.334038 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.334040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.334040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.334160 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.334544 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:24.334552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:24.334694 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.334807 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.334981 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3146,"thread_start_us":101,"threads_started":1}
I20260806 03:45:24.335177 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.320630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:53105 (local address 127.30.24.62:37183)
0806 03:45:24.320835 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.320847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.324676 (+ 3829us) server_negotiation.cc:407] Connection header received
0806 03:45:24.324811 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.324817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.324872 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.324986 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:24.326009 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.327373 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.328755 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.329251 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.329386 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.331633 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:24.331637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:24.331638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:24.331647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:24.333376 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.333853 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.333856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.333857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.333952 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.334239 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.334240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.334240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.334463 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:24.334566 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.334863 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.334990 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260806 03:45:24.336335 14852 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37183
I20260806 03:45:24.336467 14852 heartbeater.cc:464] Registering TS with master...
I20260806 03:45:24.336644 14852 heartbeater.cc:511] Master 127.30.24.62:37183 requested a full tablet report, sending...
I20260806 03:45:24.337455 14662 ts_manager.cc:194] Registered new tserver with Master: da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903)
I20260806 03:45:24.338678 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:53105
I20260806 03:45:24.339707 14852 heartbeater.cc:503] Master 127.30.24.62:37183 was elected leader, sending a full tablet report...
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.1@KRBTEST.COM
I20260806 03:45:24.347520 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.338063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:39903 (local address 127.0.0.1:38176)
0806 03:45:24.338244 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.338287 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.338363 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.338597 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.338604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.338930 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:24.338987 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.338989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.340080 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.340081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.341043 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.341046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.341533 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.341534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.341618 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.342195 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:24.342200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:24.343830 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:24.346407 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.346414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.346416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.346631 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.346859 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.346860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.346860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.346917 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.347234 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:24.347240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:24.347286 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.347350 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.347448 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260806 03:45:24.347597 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.338147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.30.24.1:39903)
0806 03:45:24.338317 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.338321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.338338 (+ 17us) server_negotiation.cc:407] Connection header received
0806 03:45:24.338473 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.338475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.338490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.338563 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:24.339175 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.339985 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.341143 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.341446 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.341509 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.343958 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:24.343966 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:24.343966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:24.343974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:24.346289 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.346718 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.346721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.346722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.346788 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.346978 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.346979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.346979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.347168 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:24.347250 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.347406 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.347505 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260806 03:45:24.347986 14786 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e937d76846bbbd30b3327c6018f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:03 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:38176
I20260806 03:45:24.348645 14861 tablet_replica.cc:330] stopping tablet replica
I20260806 03:45:24.348757 14861 raft_consensus.cc:2229] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:45:24.348836 14861 raft_consensus.cc:2258] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:45:24.349133 14861 ts_tablet_manager.cc:1905] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:45:24.351519 14861 ts_tablet_manager.cc:1918] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:45:24.351600 14861 log.cc:1196] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal/wals/59e1e937d76846bbbd30b3327c6018f3
I20260806 03:45:24.351857 14861 ts_tablet_manager.cc:1939] T 59e1e937d76846bbbd30b3327c6018f3 P da348e5b0f0744aa8409df3cc5d98e55: Deleting consensus metadata
I20260806 03:45:24.352386 14647 catalog_manager.cc:4841] TS da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903): tablet 59e1e937d76846bbbd30b3327c6018f3 (table test-table [id=5a4e3c95ff1c4f91a8665802dae167d4]) successfully deleted
W20260806 03:45:24.411861 14854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:24.412127 14854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:24.412168 14854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:24.413335 14854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:24.413410 14854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:24.413452 14854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:45:24.414911 14854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:43659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=36421
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:24.415184 14854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:24.415442 14854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:45:24.420761 14854 server_base.cc:1034] running on GCE node
W20260806 03:45:24.420733 14866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.420745 14867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.420773 14869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:24.421211 14854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:24.421473 14854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:24.422627 14854 hybrid_clock.cc:648] HybridClock initialized: now 1785987924422612 us; error 32 us; skew 500 ppm
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:24.429997 14854 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:45:24.430480 14854 webserver.cc:460] Webserver started at http://127.30.24.2:36421/ using document root <none> and password file <none>
I20260806 03:45:24.430730 14854 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:24.430826 14854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:24.432224 14854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:24.432989 14876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.433221 14854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:45:24.433331 14854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "b220811780e9474a844771ed26dd20ed"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "97a00392fb062ede72529d2047da0ea8"
server_key_iv: "d08200b0daa4ea8334b872be92af9bcf"
server_key_version: "encryptionkey@0"
I20260806 03:45:24.433702 14854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:24.463995 14854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:24.466734 14854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:24.466922 14854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:24.467193 14854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:24.467609 14854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:24.467648 14854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.467672 14854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:24.467685 14854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.474463 14854 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:43659
I20260806 03:45:24.474507 14994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:43659 every 8 connection(s)
I20260806 03:45:24.474867 14854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:45:24.476121 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14854
I20260806 03:45:24.476424 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:37389
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:37183
--webserver_port=38385
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:24.485966 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.475702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:38871 (local address 127.30.24.62:37183)
0806 03:45:24.475766 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.475768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.476027 (+ 259us) server_negotiation.cc:407] Connection header received
0806 03:45:24.476676 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.476677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.476695 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.476733 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:24.477791 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.478430 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.479759 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.479973 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.480043 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.482780 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:24.482782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:24.482782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:24.482791 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:24.484319 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.484825 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.484827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.484827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.484891 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.485192 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.485193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.485194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.485322 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:24.485399 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.485769 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.485870 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:45:24.486188 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.475697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:38871)
0806 03:45:24.475867 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.475920 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.476543 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.476892 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.476895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.477481 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:24.477683 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.477688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.478534 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.478536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.479658 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.479661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.480069 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.480070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.480225 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.480866 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:24.480877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:24.482665 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:24.484413 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.484421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.484424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.484736 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.484969 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.484970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.484971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.485106 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.485403 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:24.485410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:24.485592 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.485731 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.485907 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260806 03:45:24.486732 14995 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37183
I20260806 03:45:24.486845 14995 heartbeater.cc:464] Registering TS with master...
I20260806 03:45:24.487064 14995 heartbeater.cc:511] Master 127.30.24.62:37183 requested a full tablet report, sending...
I20260806 03:45:24.487493 14662 ts_manager.cc:194] Registered new tserver with Master: b220811780e9474a844771ed26dd20ed (127.30.24.2:43659)
I20260806 03:45:24.487844 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:38871
W20260806 03:45:24.564193 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:24.564435 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:24.564472 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:24.565701 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:24.565776 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:24.565821 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:45:24.567293 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:37389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=38385
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:24.567574 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:24.567807 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:24.573300 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.573302 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:24.573319 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:24.573645 14998 server_base.cc:1034] running on GCE node
I20260806 03:45:24.573976 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:24.574200 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:24.575353 14998 hybrid_clock.cc:648] HybridClock initialized: now 1785987924575339 us; error 28 us; skew 500 ppm
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:24.578148 14998 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:45:24.578581 14998 webserver.cc:460] Webserver started at http://127.30.24.3:38385/ using document root <none> and password file <none>
I20260806 03:45:24.578817 14998 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:24.578873 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:24.580284 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:45:24.580963 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.581171 14998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:45:24.581269 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "c3061a896ab64e04ae9479760844105f"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "9412329e4722ecd0aeafea16e85e19f5"
server_key_iv: "8a0e752eec27d55e9b5e51a0d9d51bf4"
server_key_version: "encryptionkey@0"
I20260806 03:45:24.581629 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:24.597591 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:24.599608 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:24.599807 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:24.600064 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:24.600392 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:24.600452 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.600544 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:24.600579 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:24.606529 14998 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37389
I20260806 03:45:24.606608 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37389 every 8 connection(s)
I20260806 03:45:24.607018 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:45:24 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:45:24.613902 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 14998
I20260806 03:45:24.616999 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.607893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:60047 (local address 127.30.24.62:37183)
0806 03:45:24.607974 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.607978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.608250 (+ 272us) server_negotiation.cc:407] Connection header received
0806 03:45:24.608747 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.608749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.608759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.608814 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:24.609739 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.610214 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.611192 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.611371 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.611421 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.613719 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:24.613721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:24.613721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:24.613728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:24.615523 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.615920 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.615921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.615922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.615987 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.616282 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.616283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.616284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.616404 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:24.616512 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.616816 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.616907 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 03:45:24.617177 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.607890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:60047)
0806 03:45:24.608124 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.608162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.608656 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.608868 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.608871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.609457 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:24.609657 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.609661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.610299 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.610300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.611115 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.611117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.611450 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.611451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.611553 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.612117 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:24.612125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:24.613629 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:24.615616 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.615622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.615624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.615834 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.616089 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:24.616090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:24.616090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:24.616197 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:24.616513 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:24.616521 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:24.616662 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.616774 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.616930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
I20260806 03:45:24.617671 15131 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37183
I20260806 03:45:24.617761 15131 heartbeater.cc:464] Registering TS with master...
I20260806 03:45:24.617982 15131 heartbeater.cc:511] Master 127.30.24.62:37183 requested a full tablet report, sending...
I20260806 03:45:24.618474 14662 ts_manager.cc:194] Registered new tserver with Master: c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
I20260806 03:45:24.619015 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:60047
I20260806 03:45:24.621816 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.614349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.30.24.62:37183)
0806 03:45:24.614455 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.614457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.614587 (+ 130us) server_negotiation.cc:407] Connection header received
0806 03:45:24.614853 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.614854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.614865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.614919 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:24.615700 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.616172 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.617159 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.617509 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.617583 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.619028 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:24.619030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:24.619030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:24.619036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:24.620532 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.620853 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.620854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.620854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.620909 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.621201 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.621202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.621203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.621313 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:24.621438 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.621564 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.621686 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260806 03:45:24.622599 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:45:24.625560 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.623009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.30.24.62:37183)
0806 03:45:24.623045 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.623046 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.623160 (+ 114us) server_negotiation.cc:407] Connection header received
0806 03:45:24.623221 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.623222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.623250 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.623305 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:45:24.623405 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.623845 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.624656 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.624838 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.624894 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.625347 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:45:24.625389 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 03:45:24.625520 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37046: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 03:45:24.625643 15135 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37046: token expired
W20260806 03:45:24.625871 14588 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.24.62:37183)
I20260806 03:45:24.631495 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.626152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.30.24.62:37183)
0806 03:45:24.626186 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.626188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.626196 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:45:24.626233 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.626234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.626241 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.626284 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:24.626696 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.627124 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.627746 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.627909 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.627957 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.629077 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:24.629078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:24.629078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:24.629084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:24.630553 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.630906 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.630906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.630907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.630954 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:24.631122 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:24.631123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:24.631123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:24.631215 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:24.631286 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.631385 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.631439 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 03:45:24.632794 14662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:45:24.634385 14662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:45:24.636750 14786 tablet_service.cc:1467] Processing CreateTablet for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f (DEFAULT_TABLE table=test-table [id=7297b1d5f03343caa6fd444e0f78f1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:45:24.636945 14786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:24.638458 14743 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap starting.
I20260806 03:45:24.638814 14743 tablet_bootstrap.cc:654] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:24.639873 14743 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: No bootstrap required, opened a new log
I20260806 03:45:24.639919 14743 ts_tablet_manager.cc:1397] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 03:45:24.640100 14743 raft_consensus.cc:348] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.640162 14743 raft_consensus.cc:374] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:24.640192 14743 raft_consensus.cc:729] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da348e5b0f0744aa8409df3cc5d98e55, State: Initialized, Role: FOLLOWER
I20260806 03:45:24.640275 14743 consensus_queue.cc:260] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.640388 14743 ts_tablet_manager.cc:1428] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 03:45:24.642771 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.636297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:43659 (local address 127.0.0.1:58028)
0806 03:45:24.636687 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.636697 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.636750 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.639156 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.639156 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.639159 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:24.639202 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.639203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.640329 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.640330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.641586 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.641589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.642512 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.642514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.642579 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.642598 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.642655 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.642696 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":11,"thread_start_us":114,"threads_started":1}
I20260806 03:45:24.642791 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.636441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.30.24.2:43659)
0806 03:45:24.638961 (+ 2520us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.638966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.638975 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:45:24.639015 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.639018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.639035 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.639144 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:24.639258 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.640245 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.641677 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.642412 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.642498 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.642560 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.642671 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.642715 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2474,"thread_start_us":135,"threads_started":1}
I20260806 03:45:24.643625 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.636297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37389 (local address 127.0.0.1:51844)
0806 03:45:24.636345 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.636354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.636392 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.638810 (+ 2418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.638811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.638813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:24.638910 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.638913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.641646 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.641647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.642546 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.642549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.643383 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.643384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.643457 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.643461 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.643513 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.643563 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 03:45:24.643678 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.636424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.30.24.3:37389)
0806 03:45:24.638623 (+ 2199us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.638629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.638640 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:45:24.638681 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.638683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.638701 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.638791 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:24.640347 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.641568 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.642649 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.643289 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.643369 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.643423 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.643550 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.643591 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2151,"thread_start_us":148,"threads_started":1}
I20260806 03:45:24.643653 14928 tablet_service.cc:1467] Processing CreateTablet for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f (DEFAULT_TABLE table=test-table [id=7297b1d5f03343caa6fd444e0f78f1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:45:24.644068 14928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:24.644279 15064 tablet_service.cc:1467] Processing CreateTablet for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f (DEFAULT_TABLE table=test-table [id=7297b1d5f03343caa6fd444e0f78f1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:45:24.644619 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:45:24.647179 15141 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Bootstrap starting.
I20260806 03:45:24.647389 15142 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Bootstrap starting.
I20260806 03:45:24.647873 15141 tablet_bootstrap.cc:654] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:24.648031 15142 tablet_bootstrap.cc:654] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Neither blocks nor log segments found. Creating new log.
I20260806 03:45:24.648569 15141 log.cc:824] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:24.648718 15142 log.cc:824] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:24.649315 15141 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: No bootstrap required, opened a new log
I20260806 03:45:24.649370 15141 ts_tablet_manager.cc:1397] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:45:24.649475 15142 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: No bootstrap required, opened a new log
I20260806 03:45:24.649555 15142 ts_tablet_manager.cc:1397] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:45:24.650767 15141 raft_consensus.cc:348] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.650918 15141 raft_consensus.cc:374] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:24.650940 15141 raft_consensus.cc:729] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b220811780e9474a844771ed26dd20ed, State: Initialized, Role: FOLLOWER
I20260806 03:45:24.650853 15142 raft_consensus.cc:348] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.651034 15142 raft_consensus.cc:374] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:45:24.651080 15142 raft_consensus.cc:729] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3061a896ab64e04ae9479760844105f, State: Initialized, Role: FOLLOWER
I20260806 03:45:24.651098 15141 consensus_queue.cc:260] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.651196 15142 consensus_queue.cc:260] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.651412 15141 ts_tablet_manager.cc:1428] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:45:24.651492 15142 ts_tablet_manager.cc:1428] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:45:24.651428 14995 heartbeater.cc:503] Master 127.30.24.62:37183 was elected leader, sending a full tablet report...
I20260806 03:45:24.651537 15131 heartbeater.cc:503] Master 127.30.24.62:37183 was elected leader, sending a full tablet report...
W20260806 03:45:24.725543 14996 tablet.cc:2369] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:45:24.821014 14858 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:24.821228 14858 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.821655 14858 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
W20260806 03:45:24.822540 14853 tablet.cc:2369] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:45:24.824764 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.821844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:43659 (local address 127.30.24.1:51881)
0806 03:45:24.821906 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.821918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.821976 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.822173 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.822174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.822175 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:24.822260 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.822263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.823039 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.823040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.823827 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.823829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.824421 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.824423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.824568 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.824594 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.824648 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.824685 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 03:45:24.824800 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.821899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:51881 (local address 127.30.24.2:43659)
0806 03:45:24.821944 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.821946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.821956 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:45:24.822073 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.822075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.822090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.822150 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:24.822351 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.822955 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.823929 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.824338 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.824394 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.824403 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.824679 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.824725 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:45:24.824880 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.821847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37389 (local address 127.30.24.1:53337)
0806 03:45:24.822005 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.822015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:24.822092 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:24.822263 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:24.822264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:24.822265 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:45:24.822362 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.822365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.823143 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.823144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.823949 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:24.823951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.824587 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.824589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:24.824699 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.824703 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:45:24.824764 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 03:45:24.824798 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":12,"thread_start_us":83,"threads_started":1}
I20260806 03:45:24.824918 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.821981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:53337 (local address 127.30.24.3:37389)
0806 03:45:24.822109 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.822111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.822118 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:45:24.822169 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.822170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.822183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.822247 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:45:24.822469 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.823053 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.824049 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.824455 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.824551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.824561 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.824790 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.824836 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 03:45:24.825290 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c17695ecc5f4d9fb6ec8eec5d36501f" candidate_uuid: "da348e5b0f0744aa8409df3cc5d98e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b220811780e9474a844771ed26dd20ed" is_pre_election: true
I20260806 03:45:24.825297 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c17695ecc5f4d9fb6ec8eec5d36501f" candidate_uuid: "da348e5b0f0744aa8409df3cc5d98e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3061a896ab64e04ae9479760844105f" is_pre_election: true
I20260806 03:45:24.825455 14949 raft_consensus.cc:2454] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da348e5b0f0744aa8409df3cc5d98e55 in term 0.
I20260806 03:45:24.825471 15084 raft_consensus.cc:2454] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da348e5b0f0744aa8409df3cc5d98e55 in term 0.
I20260806 03:45:24.825703 14740 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3061a896ab64e04ae9479760844105f, da348e5b0f0744aa8409df3cc5d98e55; no voters:
I20260806 03:45:24.825855 14858 raft_consensus.cc:2781] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:45:24.825918 14858 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:45:24.825935 14858 raft_consensus.cc:3037] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:24.826695 14858 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.826841 14858 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 1 election: Requested vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
I20260806 03:45:24.827025 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c17695ecc5f4d9fb6ec8eec5d36501f" candidate_uuid: "da348e5b0f0744aa8409df3cc5d98e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3061a896ab64e04ae9479760844105f"
I20260806 03:45:24.827018 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c17695ecc5f4d9fb6ec8eec5d36501f" candidate_uuid: "da348e5b0f0744aa8409df3cc5d98e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b220811780e9474a844771ed26dd20ed"
I20260806 03:45:24.827126 15084 raft_consensus.cc:3037] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:24.827121 14949 raft_consensus.cc:3037] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:45:24.827831 14949 raft_consensus.cc:2454] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da348e5b0f0744aa8409df3cc5d98e55 in term 1.
I20260806 03:45:24.827999 15084 raft_consensus.cc:2454] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da348e5b0f0744aa8409df3cc5d98e55 in term 1.
I20260806 03:45:24.828013 14738 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b220811780e9474a844771ed26dd20ed, da348e5b0f0744aa8409df3cc5d98e55; no voters:
I20260806 03:45:24.828212 14858 raft_consensus.cc:2781] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:45:24.828379 14858 raft_consensus.cc:686] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 LEADER]: Becoming Leader. State: Replica: da348e5b0f0744aa8409df3cc5d98e55, State: Running, Role: LEADER
I20260806 03:45:24.828492 14858 consensus_queue.cc:237] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:24.829361 14661 catalog_manager.cc:5495] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 reported cstate change: term changed from 0 to 1, leader changed from <none> to da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "da348e5b0f0744aa8409df3cc5d98e55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } health_report { overall_health: HEALTHY } } }
W20260806 03:45:24.857714 15132 tablet.cc:2369] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:45:24.868007 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.865548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.30.24.1:39903)
0806 03:45:24.865668 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:24.865669 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:24.865677 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:45:24.865716 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:24.865716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:24.865737 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:24.865807 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:45:24.865934 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.866423 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.867143 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:24.867331 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:24.867379 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:24.867567 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:45:24.867673 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:45:24.867792 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:24.867838 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 03:45:24.867867 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260806 03:45:24.869608 14949 raft_consensus.cc:1261] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Refusing update from remote peer da348e5b0f0744aa8409df3cc5d98e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:45:24.869608 15084 raft_consensus.cc:1261] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Refusing update from remote peer da348e5b0f0744aa8409df3cc5d98e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:45:24.869968 15148 consensus_queue.cc:1035] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:45:24.870108 14858 consensus_queue.cc:1035] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:45:24.872597 15152 mvcc.cc:205] Tried to move back new op lower bound from 7315406540263104512 to 7315406540098260992. Current Snapshot: MvccSnapshot[applied={T|T < 7315406540263104512}]
I20260806 03:45:24.873356 15153 mvcc.cc:205] Tried to move back new op lower bound from 7315406540263104512 to 7315406540098260992. Current Snapshot: MvccSnapshot[applied={T|T < 7315406540263104512}]
I20260806 03:45:24.875497 14661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:45:24.875586 14661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:45:24.876320 14661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Sending DeleteTablet for 3 replicas of tablet 5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:24.876729 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14701
I20260806 03:45:24.876718 15064 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:24 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:51844
I20260806 03:45:24.876756 14928 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:24 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:58028
I20260806 03:45:24.876924 14786 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:45:24 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:38176
I20260806 03:45:24.877897 15162 tablet_replica.cc:330] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: stopping tablet replica
I20260806 03:45:24.877977 15162 raft_consensus.cc:2229] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:45:24.878053 15162 raft_consensus.cc:2258] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:45:24.878268 15163 tablet_replica.cc:330] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: stopping tablet replica
I20260806 03:45:24.878363 15163 raft_consensus.cc:2229] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:45:24.878455 15163 raft_consensus.cc:2258] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:45:24.878504 15162 ts_tablet_manager.cc:1905] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:45:24.878906 15163 ts_tablet_manager.cc:1905] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:45:24.880389 15162 ts_tablet_manager.cc:1918] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:45:24.880496 15162 log.cc:1196] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:24.880673 15163 ts_tablet_manager.cc:1918] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:45:24.880776 15163 log.cc:1196] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:24.880873 15162 ts_tablet_manager.cc:1939] T 5c17695ecc5f4d9fb6ec8eec5d36501f P b220811780e9474a844771ed26dd20ed: Deleting consensus metadata
I20260806 03:45:24.881085 15163 ts_tablet_manager.cc:1939] T 5c17695ecc5f4d9fb6ec8eec5d36501f P c3061a896ab64e04ae9479760844105f: Deleting consensus metadata
I20260806 03:45:24.881312 14647 catalog_manager.cc:4841] TS b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): tablet 5c17695ecc5f4d9fb6ec8eec5d36501f (table test-table [id=7297b1d5f03343caa6fd444e0f78f1a7]) successfully deleted
I20260806 03:45:24.881595 14649 catalog_manager.cc:4841] TS c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): tablet 5c17695ecc5f4d9fb6ec8eec5d36501f (table test-table [id=7297b1d5f03343caa6fd444e0f78f1a7]) successfully deleted
W20260806 03:45:24.884231 14647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:39903 (error 108)
I20260806 03:45:24.884671 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14854
I20260806 03:45:24.884789 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:24.884619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:39903 (local address 127.0.0.1:38196)
0806 03:45:24.884682 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:24.884751 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:39903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 03:45:24.885640 14647 catalog_manager.cc:4584] TS da348e5b0f0744aa8409df3cc5d98e55 (127.30.24.1:39903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c17695ecc5f4d9fb6ec8eec5d36501f: Network error: Client connection negotiation failed: client connection to 127.30.24.1:39903: connect: Connection refused (error 111)
I20260806 03:45:24.893035 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14998
I20260806 03:45:24.898841 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 14629
I20260806 03:45:44.905431 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37183
--webserver_interface=127.30.24.62
--webserver_port=40301
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:44.990401 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:44.990662 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:44.990702 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:44.991860 15165 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 03:45:44.991933 15165 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 03:45:44.991954 15165 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 03:45:44.991974 15165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:45:44.992003 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:44.992034 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:44.992063 15165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:45:44.992094 15165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:45:44.993613 15165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=40301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:44.993907 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:44.994146 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:44.999596 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:44.999666 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:44.999588 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:44.999681 15165 server_base.cc:1034] running on GCE node
I20260806 03:45:45.000128 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:45.000401 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:45.001549 15165 hybrid_clock.cc:648] HybridClock initialized: now 1785987945001524 us; error 44 us; skew 500 ppm
Aug 06 03:45:45 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987945, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:45.009346 15165 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:45:45.009990 15165 webserver.cc:460] Webserver started at http://127.30.24.62:40301/ using document root <none> and password file <none>
I20260806 03:45:45.010254 15165 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:45.010304 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:45.011812 15165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:45.012629 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.012814 15165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:45:45.012924 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "f42699a521e54df3b62a9b7ff9bde11e"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "e511ab112f3b881bffe22eac11863aaa"
server_key_iv: "181d2a202645048b1f7edef060265869"
server_key_version: "encryptionkey@0"
I20260806 03:45:45.013329 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:45.034444 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:45.037036 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:45.037284 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:45.041558 15165 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:37183
I20260806 03:45:45.041626 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:37183 every 8 connection(s)
I20260806 03:45:45.042086 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:45:45.043630 15236 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:45:45.044342 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap starting.
I20260806 03:45:45.052295 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15165
I20260806 03:45:45.052641 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:39903
--local_ip_for_outbound_sockets=127.30.24.1
--tserver_master_addrs=127.30.24.62:37183
--webserver_port=35757
--webserver_interface=127.30.24.1
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:45:45.053272 15236 log.cc:824] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:45.055748 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:45:45.056072 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Bootstrap complete.
I20260806 03:45:45.058491 15236 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:45.058749 15236 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42699a521e54df3b62a9b7ff9bde11e, State: Initialized, Role: FOLLOWER
I20260806 03:45:45.058938 15236 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:45.059017 15236 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:45:45.059072 15236 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:45:45.059154 15236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:45:45.060156 15236 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:45.060279 15236 leader_election.cc:302] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42699a521e54df3b62a9b7ff9bde11e; no voters:
I20260806 03:45:45.060503 15236 leader_election.cc:288] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 03:45:45.060539 15240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:45:45.060668 15240 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [term 3 LEADER]: Becoming Leader. State: Replica: f42699a521e54df3b62a9b7ff9bde11e, State: Running, Role: LEADER
I20260806 03:45:45.060794 15236 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:45:45.060835 15240 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } }
I20260806 03:45:45.061206 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f42699a521e54df3b62a9b7ff9bde11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } } }
I20260806 03:45:45.061228 15242 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42699a521e54df3b62a9b7ff9bde11e. Latest consensus state: current_term: 3 leader_uuid: "f42699a521e54df3b62a9b7ff9bde11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42699a521e54df3b62a9b7ff9bde11e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37183 } } }
I20260806 03:45:45.061313 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: This master's current role is: LEADER
I20260806 03:45:45.061321 15242 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e [sys.catalog]: This master's current role is: LEADER
I20260806 03:45:45.061556 15247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:45:45.062713 15247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:45:45.062860 15247 catalog_manager.cc:1259] Loaded cluster ID: 11e4fa43d31b4ee9b9faf670c9b06d00
I20260806 03:45:45.062934 15247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:45:45.064306 15247 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:45:45.064524 15247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Loaded TSK: 0
I20260806 03:45:45.064579 15247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f42699a521e54df3b62a9b7ff9bde11e: Loaded TSK: 1
F20260806 03:45:45.064812 15247 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f998d39f9ad google::LogMessage::Fail() at ??:0
@ 0x7f998d3a288a google::LogMessage::SendToLog() at ??:0
@ 0x7f998d39f4de google::LogMessage::Flush() at ??:0
@ 0x7f998d3a0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f998fb0c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f998fb09d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f99952f91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f99952f948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f99952f9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f99952bfc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f99952c9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f998f26d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f998f262954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f99900c66db start_thread at ??:0
@ 0x7f998be9d61f clone at ??:0
W20260806 03:45:45.167095 15239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:45.167387 15239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:45.167430 15239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:45.169185 15239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:45.169258 15239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:45.169301 15239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:45:45.171491 15239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:39903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=35757
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:45.171765 15239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:45.172049 15239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:45.178437 15280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:45.178432 15277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:45.178655 15239 server_base.cc:1034] running on GCE node
W20260806 03:45:45.178439 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:45.178969 15239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:45.179229 15239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:45.180397 15239 hybrid_clock.cc:648] HybridClock initialized: now 1785987945180382 us; error 31 us; skew 500 ppm
Aug 06 03:45:45 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987945, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:45.183079 15239 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:45:45.183589 15239 webserver.cc:460] Webserver started at http://127.30.24.1:35757/ using document root <none> and password file <none>
I20260806 03:45:45.183864 15239 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:45.183938 15239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:45.185498 15239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:45.186262 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.186462 15239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:45.186581 15239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "da348e5b0f0744aa8409df3cc5d98e55"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "2066dccf44fc19d4096bcb61c1d5ec6f"
server_key_iv: "56edfe836314b471c3264f483f60b6b9"
server_key_version: "encryptionkey@0"
I20260806 03:45:45.186962 15239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:45.217481 15239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:45.220772 15239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:45.220929 15239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:45.221181 15239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:45.221810 15294 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 03:45:45.223918 15239 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 03:45:45.224004 15239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 03:45:45.224047 15239 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 03:45:45.225050 15239 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 03:45:45.225100 15239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260806 03:45:45.225284 15294 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap starting.
I20260806 03:45:45.232640 15239 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:39903
I20260806 03:45:45.232688 15404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:39903 every 8 connection(s)
I20260806 03:45:45.233100 15239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:45:45.236708 15294 log.cc:824] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Log is configured to *not* fsync() on all Append() calls
I20260806 03:45:45.239637 15294 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:45:45.239919 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15239
I20260806 03:45:45.240025 15294 tablet_bootstrap.cc:492] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Bootstrap complete.
I20260806 03:45:45.240239 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:43659
--local_ip_for_outbound_sockets=127.30.24.2
--tserver_master_addrs=127.30.24.62:37183
--webserver_port=36421
--webserver_interface=127.30.24.2
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:45:45.241377 15294 ts_tablet_manager.cc:1397] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260806 03:45:45.242972 15294 raft_consensus.cc:348] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:45.243319 15294 raft_consensus.cc:729] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da348e5b0f0744aa8409df3cc5d98e55, State: Initialized, Role: FOLLOWER
I20260806 03:45:45.243497 15294 consensus_queue.cc:260] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:45.243803 15294 ts_tablet_manager.cc:1428] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260806 03:45:45.296186 15407 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:45:45.233862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:50887)
0806 03:45:45.234045 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.234090 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:45.234571 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:45.295964 (+ 61393us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
W20260806 03:45:45.296507 15405 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:37183 (0 consecutive failures): Network error: Failed to ping master at 127.30.24.62:37183: Client connection negotiation failed: client connection to 127.30.24.62:37183: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 03:45:45.296898 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.296770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:59855)
0806 03:45:45.296826 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.296869 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 03:45:45.297309 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.297217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:38689)
0806 03:45:45.297269 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.297294 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 03:45:45.297447 15405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:45:45.329365 15409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:45.329618 15409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:45.329667 15409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:45.330863 15409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:45.330935 15409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:45.330982 15409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:45:45.332576 15409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:43659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=36421
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:45.332868 15409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:45.333124 15409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:45.338528 15416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:45.338590 15418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:45.338560 15415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:45.338855 15409 server_base.cc:1034] running on GCE node
I20260806 03:45:45.339077 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:45.339306 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:45.340516 15409 hybrid_clock.cc:648] HybridClock initialized: now 1785987945340495 us; error 31 us; skew 500 ppm
Aug 06 03:45:45 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987945, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:45.343158 15409 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:45:45.343593 15409 webserver.cc:460] Webserver started at http://127.30.24.2:36421/ using document root <none> and password file <none>
I20260806 03:45:45.343864 15409 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:45.343940 15409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:45.345469 15409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:45.346127 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.346321 15409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:45.346432 15409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "b220811780e9474a844771ed26dd20ed"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "97a00392fb062ede72529d2047da0ea8"
server_key_iv: "d08200b0daa4ea8334b872be92af9bcf"
server_key_version: "encryptionkey@0"
I20260806 03:45:45.346800 15409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:45.357733 15409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:45.360227 15409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:45.360421 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:45.360759 15409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:45.361099 15409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:45.361155 15409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.361203 15409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:45.361233 15409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.367460 15409 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:43659
I20260806 03:45:45.367542 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:43659 every 8 connection(s)
I20260806 03:45:45.367919 15409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:45:45.369084 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.368689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:36483)
0806 03:45:45.368858 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.368933 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260806 03:45:45.369535 15541 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:37183 (0 consecutive failures): Network error: Failed to ping master at 127.30.24.62:37183: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
I20260806 03:45:45.369858 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.369731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:53975)
0806 03:45:45.369805 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.369836 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 03:45:45.370185 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.370097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:35087)
0806 03:45:45.370140 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.370162 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 03:45:45.370321 15541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:45:45.377571 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15409
I20260806 03:45:45.377828 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:37389
--local_ip_for_outbound_sockets=127.30.24.3
--tserver_master_addrs=127.30.24.62:37183
--webserver_port=38385
--webserver_interface=127.30.24.3
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:45:45.463469 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:45:45.463692 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:45:45.463718 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:45:45.464890 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:45:45.464939 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:45:45.464967 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:45:45.466415 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:37389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=38385
--tserver_master_addrs=127.30.24.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:45:45.466619 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:45:45.466805 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:45:45.471887 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:45.471891 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:45:45.471909 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:45:45.472213 15544 server_base.cc:1034] running on GCE node
I20260806 03:45:45.472483 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:45:45.472743 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:45:45.473912 15544 hybrid_clock.cc:648] HybridClock initialized: now 1785987945473898 us; error 44 us; skew 500 ppm
Aug 06 03:45:45 dist-test-slave-10pc krb5kdc[14069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987945, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:45:45.476586 15544 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:45:45.476982 15544 webserver.cc:460] Webserver started at http://127.30.24.3:38385/ using document root <none> and password file <none>
I20260806 03:45:45.477183 15544 fs_manager.cc:359] Metadata directory not provided
I20260806 03:45:45.477226 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:45:45.478560 15544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:45:45.479138 15560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.479302 15544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:45:45.479378 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "c3061a896ab64e04ae9479760844105f"
format_stamp: "Formatted at 2026-08-06 03:45:03 on dist-test-slave-10pc"
server_key: "9412329e4722ecd0aeafea16e85e19f5"
server_key_iv: "8a0e752eec27d55e9b5e51a0d9d51bf4"
server_key_version: "encryptionkey@0"
I20260806 03:45:45.479760 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:45:45.501291 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:45:45.503844 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:45:45.504019 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:45:45.504295 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:45:45.504669 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:45:45.504729 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.504766 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:45:45.504798 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:45:45.510802 15544 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:37389
I20260806 03:45:45.510915 15675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:37389 every 8 connection(s)
I20260806 03:45:45.511266 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:45:45.512588 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.512114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:36257)
0806 03:45:45.512279 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.512358 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260806 03:45:45.513054 15676 heartbeater.cc:644] Failed to heartbeat to 127.30.24.62:37183 (0 consecutive failures): Network error: Failed to ping master at 127.30.24.62:37183: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
I20260806 03:45:45.513353 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.513266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:44873)
0806 03:45:45.513308 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.513335 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 03:45:45.513664 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:45.513549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:44055)
0806 03:45:45.513587 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:45.513610 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 03:45:45.513835 15676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:45:45.513857 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15544
I20260806 03:45:46.298013 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.297783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:38079)
0806 03:45:46.297912 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:46.297948 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260806 03:45:46.370918 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.370686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:60309)
0806 03:45:46.370829 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:46.370865 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260806 03:45:46.514330 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.514136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:44235)
0806 03:45:46.514248 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:46.514281 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260806 03:45:46.649233 15683 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:46.649433 15683 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:46.649859 15683 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
Aug 06 03:45:46 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987945, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.2@KRBTEST.COM
Aug 06 03:45:46 dist-test-slave-10pc krb5kdc[14069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987945, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.3@KRBTEST.COM
I20260806 03:45:46.663007 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.650164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:48211 (local address 127.30.24.3:37389)
0806 03:45:46.650298 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:46.650311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:46.650349 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:45:46.650949 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:46.650952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:46.651010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:46.651163 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:46.652217 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.653650 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.655222 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.655884 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.656067 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:46.659492 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:46.659497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:46.659497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:46.659515 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:46.661350 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:46.661785 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:46.661789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:46.661789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:46.661921 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:46.662167 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:46.662167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:46.662168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:46.662432 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:46.662516 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:46.662745 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 03:45:46.662881 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260806 03:45:46.662995 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.650050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:43659 (local address 127.30.24.1:53479)
0806 03:45:46.650189 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:46.650200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:46.650258 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:46.651265 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:46.651268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:46.651761 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:46.652077 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:46.652085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.653778 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.653779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:46.655100 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:46.655103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.655996 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.655996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:46.656132 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:46.656908 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:46.656920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:46.658824 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:46.661177 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:46.661185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:46.661187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:46.661519 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:46.661799 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:46.661800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:46.661800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:46.661952 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:46.662415 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:46.662422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:46.662569 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 03:45:46.662691 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 03:45:46.662893 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":72,"thread_start_us":80,"threads_started":1}
I20260806 03:45:46.663050 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.650182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:53479 (local address 127.30.24.2:43659)
0806 03:45:46.650350 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 03:45:46.650357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:45:46.650385 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:45:46.650948 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:45:46.650952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:45:46.651010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:45:46.651163 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:45:46.652217 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.653659 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.655222 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.655884 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.656012 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:46.658966 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:45:46.658972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:45:46.658973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:45:46.658988 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:45:46.661079 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:46.661602 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:46.661606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:46.661607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:46.661726 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:45:46.662022 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:45:46.662023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:45:46.662024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:45:46.662341 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:45:46.662402 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:45:46.662774 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 03:45:46.662930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
I20260806 03:45:46.663520 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:46.650029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:37389 (local address 127.30.24.1:48211)
0806 03:45:46.650089 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:46.650101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:45:46.650157 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:45:46.651265 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:45:46.651268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:45:46.651762 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:45:46.652077 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:46.652085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.653792 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.653793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:46.655100 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:45:46.655103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:45:46.655994 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:45:46.655995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:45:46.656143 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:45:46.656917 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:45:46.656922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:45:46.659354 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:45:46.661443 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:46.661444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:46.661445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:46.661675 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:46.662021 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:45:46.662022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:45:46.662022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:45:46.662095 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:45:46.662509 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:45:46.662510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:45:46.662570 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 03:45:46.662655 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 03:45:46.662981 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":215}
W20260806 03:45:46.663992 15289 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
W20260806 03:45:46.664295 15291 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:46.664366 15291 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da348e5b0f0744aa8409df3cc5d98e55; no voters: b220811780e9474a844771ed26dd20ed, c3061a896ab64e04ae9479760844105f
I20260806 03:45:46.664520 15683 raft_consensus.cc:2726] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:45:47.298758 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:47.298509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:50259)
0806 03:45:47.298648 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:47.298699 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260806 03:45:47.371688 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:47.371424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:53421)
0806 03:45:47.371580 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:47.371644 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260806 03:45:47.515081 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:47.514856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:56481)
0806 03:45:47.515002 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:47.515040 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260806 03:45:48.299485 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:48.299266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:34723)
0806 03:45:48.299395 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:48.299434 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260806 03:45:48.372392 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:48.372170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:57267)
0806 03:45:48.372317 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:48.372347 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260806 03:45:48.381024 15692 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:48.381114 15692 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:48.381291 15692 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
W20260806 03:45:48.381767 15289 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
W20260806 03:45:48.381865 15291 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:48.381929 15289 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da348e5b0f0744aa8409df3cc5d98e55; no voters: b220811780e9474a844771ed26dd20ed, c3061a896ab64e04ae9479760844105f
I20260806 03:45:48.382043 15692 raft_consensus.cc:2726] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:45:48.515831 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:48.515598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:39989)
0806 03:45:48.515744 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:48.515786 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260806 03:45:49.300141 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:49.299946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:45979)
0806 03:45:49.300061 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:49.300092 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260806 03:45:49.373101 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:49.372860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:49247)
0806 03:45:49.373014 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:49.373057 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260806 03:45:49.516600 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:49.516353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:41289)
0806 03:45:49.516508 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:49.516553 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260806 03:45:50.300925 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:50.300675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:57503)
0806 03:45:50.300831 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:50.300874 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260806 03:45:50.373834 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:50.373553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:39673)
0806 03:45:50.373746 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:50.373789 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":111,"threads_started":1}
I20260806 03:45:50.517326 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:50.517107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:42807)
0806 03:45:50.517238 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:50.517281 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260806 03:45:51.301586 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:51.301376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:48557)
0806 03:45:51.301500 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:51.301538 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260806 03:45:51.374476 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:51.374265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:49417)
0806 03:45:51.374390 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:51.374430 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260806 03:45:51.479377 15702 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:51.479483 15702 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:51.479664 15702 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
W20260806 03:45:51.480111 15291 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
W20260806 03:45:51.480195 15289 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:51.480276 15289 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da348e5b0f0744aa8409df3cc5d98e55; no voters: b220811780e9474a844771ed26dd20ed, c3061a896ab64e04ae9479760844105f
I20260806 03:45:51.480515 15702 raft_consensus.cc:2726] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:45:51.518049 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:51.517825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:54451)
0806 03:45:51.517966 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:51.518005 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260806 03:45:52.302304 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:52.302064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:37131)
0806 03:45:52.302224 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:52.302255 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260806 03:45:52.375154 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:52.374911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:44977)
0806 03:45:52.375064 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:52.375106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260806 03:45:52.518873 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:52.518555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:35793)
0806 03:45:52.518781 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:52.518824 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260806 03:45:53.303092 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:53.302840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:45393)
0806 03:45:53.302993 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:53.303043 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260806 03:45:53.375970 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:53.375699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:50721)
0806 03:45:53.375862 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:53.375908 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260806 03:45:53.519651 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:53.519402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:55439)
0806 03:45:53.519561 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:53.519603 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260806 03:45:54.303901 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:54.303638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:34055)
0806 03:45:54.303815 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:54.303853 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":124,"threads_started":1}
I20260806 03:45:54.376763 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:54.376519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:48191)
0806 03:45:54.376668 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:54.376715 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260806 03:45:54.520426 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:54.520188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:38403)
0806 03:45:54.520338 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:54.520376 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260806 03:45:54.592495 15713 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:54.592586 15713 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:54.592756 15713 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
W20260806 03:45:54.593155 15289 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
W20260806 03:45:54.593222 15291 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:54.593251 15289 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da348e5b0f0744aa8409df3cc5d98e55; no voters: b220811780e9474a844771ed26dd20ed, c3061a896ab64e04ae9479760844105f
I20260806 03:45:54.593330 15713 raft_consensus.cc:2726] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:45:55.304929 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:55.304682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:43671)
0806 03:45:55.304817 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:55.304879 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260806 03:45:55.377435 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:55.377222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:48861)
0806 03:45:55.377350 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:55.377392 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260806 03:45:55.521136 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:55.520912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:39615)
0806 03:45:55.521052 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:55.521088 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260806 03:45:56.305637 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:56.305401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:58331)
0806 03:45:56.305537 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:56.305588 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260806 03:45:56.378129 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:56.377929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:44537)
0806 03:45:56.378051 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:56.378081 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260806 03:45:56.521826 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:56.521605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:33627)
0806 03:45:56.521735 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:56.521777 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260806 03:45:57.306342 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:57.306106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:54463)
0806 03:45:57.306252 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:57.306291 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260806 03:45:57.378979 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:57.378665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:57081)
0806 03:45:57.378879 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:57.378928 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260806 03:45:57.522583 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:57.522323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:59825)
0806 03:45:57.522493 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:57.522536 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260806 03:45:58.307231 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:58.306888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:52959)
0806 03:45:58.307072 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:58.307118 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20260806 03:45:58.379820 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:58.379533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:60599)
0806 03:45:58.379722 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:58.379769 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260806 03:45:58.488636 15725 raft_consensus.cc:482] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:45:58.488745 15725 raft_consensus.cc:504] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b220811780e9474a844771ed26dd20ed" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 43659 } } peers { permanent_uuid: "c3061a896ab64e04ae9479760844105f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 37389 } } peers { permanent_uuid: "da348e5b0f0744aa8409df3cc5d98e55" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 39903 } }
I20260806 03:45:58.488951 15725 leader_election.cc:288] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b220811780e9474a844771ed26dd20ed (127.30.24.2:43659), c3061a896ab64e04ae9479760844105f (127.30.24.3:37389)
W20260806 03:45:58.489356 15291 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3061a896ab64e04ae9479760844105f (127.30.24.3:37389): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
W20260806 03:45:58.489440 15289 leader_election.cc:341] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b220811780e9474a844771ed26dd20ed (127.30.24.2:43659): Not found: Tablet not found: 5c17695ecc5f4d9fb6ec8eec5d36501f
I20260806 03:45:58.489480 15289 leader_election.cc:302] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da348e5b0f0744aa8409df3cc5d98e55; no voters: b220811780e9474a844771ed26dd20ed, c3061a896ab64e04ae9479760844105f
I20260806 03:45:58.489557 15725 raft_consensus.cc:2726] T 5c17695ecc5f4d9fb6ec8eec5d36501f P da348e5b0f0744aa8409df3cc5d98e55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:45:58.523350 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:58.523112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:36585)
0806 03:45:58.523250 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:58.523287 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260806 03:45:59.308095 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:59.307764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:49517)
0806 03:45:59.307985 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:59.308036 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":131,"threads_started":1}
I20260806 03:45:59.380643 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:59.380382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:39219)
0806 03:45:59.380554 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:59.380595 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":127,"threads_started":1}
I20260806 03:45:59.524104 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:45:59.523862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:43749)
0806 03:45:59.524021 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 03:45:59.524055 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260806 03:46:00.308902 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:00.308700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.1:55911)
0806 03:46:00.308821 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:00.308855 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260806 03:46:00.381441 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:00.381219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.2:41629)
0806 03:46:00.381361 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:00.381395 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260806 03:46:00.516196 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 15239
I20260806 03:46:00.522637 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 15409
I20260806 03:46:00.524917 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:00.524715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37183 (local address 127.30.24.3:36407)
0806 03:46:00.524843 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:00.524876 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.62:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260806 03:46:00.527814 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 15544
2026-08-06T03:46:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58794 ms)
[----------] 30 tests from SecurityITest (236104 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:00 dist-test-slave-10pc krb5kdc[15735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:00 dist-test-slave-10pc krb5kdc[15735](info): set up 2 sockets
Aug 06 03:46:00 dist-test-slave-10pc krb5kdc[15735](info): commencing operation
krb5kdc: starting...
W20260806 03:46:01.581938 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:01 dist-test-slave-10pc krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:01.711292 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37151
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:37151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:01.796761 15748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:01.796962 15748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:01.796983 15748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:01.798135 15748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:01.798178 15748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:01.798193 15748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:01.798229 15748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:01.798256 15748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:01.799755 15748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:37151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:01.800014 15748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:01.800259 15748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:01.805821 15756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:01.805958 15754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:01.806016 15748 server_base.cc:1034] running on GCE node
W20260806 03:46:01.805887 15753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:01.806341 15748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:01.806702 15748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:01.807869 15748 hybrid_clock.cc:648] HybridClock initialized: now 1785987961807850 us; error 36 us; skew 500 ppm
Aug 06 03:46:01 dist-test-slave-10pc krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:01.810425 15748 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:01.810879 15748 webserver.cc:460] Webserver started at http://127.30.24.62:36841/ using document root <none> and password file <none>
I20260806 03:46:01.811133 15748 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:01.811203 15748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:01.811350 15748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:01.812429 15748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "1e1586edbd4c4f72a2332342c14e62a3"
format_stamp: "Formatted at 2026-08-06 03:46:01 on dist-test-slave-10pc"
server_key: "b8230c766c3174d9f73592739929b2fe"
server_key_iv: "ae3b9e84a9fbbc8735f9e3c925473058"
server_key_version: "encryptionkey@0"
I20260806 03:46:01.812873 15748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "1e1586edbd4c4f72a2332342c14e62a3"
format_stamp: "Formatted at 2026-08-06 03:46:01 on dist-test-slave-10pc"
server_key: "b8230c766c3174d9f73592739929b2fe"
server_key_iv: "ae3b9e84a9fbbc8735f9e3c925473058"
server_key_version: "encryptionkey@0"
I20260806 03:46:01.814224 15748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:01.814996 15763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:01.815174 15748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:01.815266 15748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "1e1586edbd4c4f72a2332342c14e62a3"
format_stamp: "Formatted at 2026-08-06 03:46:01 on dist-test-slave-10pc"
server_key: "b8230c766c3174d9f73592739929b2fe"
server_key_iv: "ae3b9e84a9fbbc8735f9e3c925473058"
server_key_version: "encryptionkey@0"
I20260806 03:46:01.815354 15748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:01.823391 15748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:01.825769 15748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:01.825937 15748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:01.829871 15748 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:37151
I20260806 03:46:01.829906 15815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:37151 every 8 connection(s)
I20260806 03:46:01.830264 15748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:01.831166 15816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:01.833554 15816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3: Bootstrap starting.
I20260806 03:46:01.834142 15816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:01.834444 15816 log.cc:824] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:01.835140 15816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3: No bootstrap required, opened a new log
I20260806 03:46:01.836856 15816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1586edbd4c4f72a2332342c14e62a3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37151 } }
I20260806 03:46:01.836995 15816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:01.837018 15816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1586edbd4c4f72a2332342c14e62a3, State: Initialized, Role: FOLLOWER
I20260806 03:46:01.837244 15816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1586edbd4c4f72a2332342c14e62a3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37151 } }
I20260806 03:46:01.837330 15816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:01.837321 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15748
I20260806 03:46:01.837376 15816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:01.837424 15816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:01.837456 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:01.837653 30816 external_mini_cluster.cc:1428] Setting key 9209265c461b5ef3dd1fb859b30398d4
I20260806 03:46:01.838071 15816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1586edbd4c4f72a2332342c14e62a3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37151 } }
I20260806 03:46:01.838223 15816 leader_election.cc:302] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e1586edbd4c4f72a2332342c14e62a3; no voters:
I20260806 03:46:01.838486 15816 leader_election.cc:288] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:01.838510 15822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:01.838622 15822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1e1586edbd4c4f72a2332342c14e62a3, State: Running, Role: LEADER
I20260806 03:46:01.838775 15816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:01.838759 15822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1586edbd4c4f72a2332342c14e62a3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37151 } }
I20260806 03:46:01.839397 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e1586edbd4c4f72a2332342c14e62a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1586edbd4c4f72a2332342c14e62a3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37151 } } }
I20260806 03:46:01.839499 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:01.840010 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e1586edbd4c4f72a2332342c14e62a3. Latest consensus state: current_term: 1 leader_uuid: "1e1586edbd4c4f72a2332342c14e62a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1586edbd4c4f72a2332342c14e62a3" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37151 } } }
I20260806 03:46:01.840087 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:01.840294 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:01.841015 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:01.842527 15831 catalog_manager.cc:1347] Generated new cluster ID: 55c36da165ae40d98902ee25190cf226
I20260806 03:46:01.842583 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:46:01 dist-test-slave-10pc krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:01.850015 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:01.838159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.30.24.62:37151)
0806 03:46:01.838357 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:01.838370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:01.838400 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:46:01.838493 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:01.838498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:01.838561 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:01.839532 (+ 971us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:01.839633 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:01.841138 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:01.842217 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:01.842804 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:01.843245 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:01.845841 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:01.845845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:01.845845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:01.845857 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:01.847825 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:01.848250 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:01.848253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:01.848254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:01.848368 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:01.848650 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:01.848651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:01.848651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:01.849021 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:01.849573 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:01.849651 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 03:46:01.849795 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260806 03:46:01.858544 15831 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:01.859206 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:01.867754 15831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3: Generated new TSK 0
I20260806 03:46:01.867949 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:46:01.897396 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37151
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:01.982548 15845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:01.982758 15845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:01.982784 15845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:01.983911 15845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:01.983951 15845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:01.983978 15845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:46:01.985486 15845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:01.985695 15845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:01.985893 15845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:01.991209 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:01.991290 15845 server_base.cc:1034] running on GCE node
W20260806 03:46:01.991210 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:01.991230 15851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:01.991700 15845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:01.992012 15845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:01.993182 15845 hybrid_clock.cc:648] HybridClock initialized: now 1785987961993181 us; error 38 us; skew 500 ppm
Aug 06 03:46:01 dist-test-slave-10pc krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:01.995834 15845 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:46:01.996235 15845 webserver.cc:460] Webserver started at http://127.30.24.1:46675/ using document root <none> and password file <none>
I20260806 03:46:01.996438 15845 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:01.996541 15845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:01.996642 15845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:01.997696 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "d428c26828ae4ded957277e30a80eea9"
format_stamp: "Formatted at 2026-08-06 03:46:01 on dist-test-slave-10pc"
server_key: "8c7347d11c5ec7d1a40239ced84ac864"
server_key_iv: "098bf254518cfb27ef95fde08777a486"
server_key_version: "encryptionkey@0"
I20260806 03:46:01.998014 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "d428c26828ae4ded957277e30a80eea9"
format_stamp: "Formatted at 2026-08-06 03:46:01 on dist-test-slave-10pc"
server_key: "8c7347d11c5ec7d1a40239ced84ac864"
server_key_iv: "098bf254518cfb27ef95fde08777a486"
server_key_version: "encryptionkey@0"
I20260806 03:46:01.999306 15845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:02.000031 15861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.000207 15845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:02.000303 15845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "d428c26828ae4ded957277e30a80eea9"
format_stamp: "Formatted at 2026-08-06 03:46:01 on dist-test-slave-10pc"
server_key: "8c7347d11c5ec7d1a40239ced84ac864"
server_key_iv: "098bf254518cfb27ef95fde08777a486"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.000380 15845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:02.010699 15845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:02.012629 15845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:02.012802 15845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:02.013038 15845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:02.013396 15845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:02.013453 15845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.013495 15845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:02.013525 15845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.019321 15845 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:40145
I20260806 03:46:02.019403 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:40145 every 8 connection(s)
I20260806 03:46:02.019744 15845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:46:02.023100 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15845
I20260806 03:46:02.023211 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:46:02.023434 30816 external_mini_cluster.cc:1428] Setting key a6596dfb3674edfb8e2813e4f260e24e
Aug 06 03:46:02 dist-test-slave-10pc krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:02.030774 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.020566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:55525 (local address 127.30.24.62:37151)
0806 03:46:02.020650 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.020652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.020920 (+ 268us) server_negotiation.cc:407] Connection header received
0806 03:46:02.021489 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.021490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.021506 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.021562 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:02.022716 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.023273 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.024600 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.024823 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.024901 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.027156 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:02.027158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:02.027158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:02.027174 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:02.028991 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.029529 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.029530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.029530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.029594 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.029960 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.029961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.029962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.030135 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:02.030203 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.030610 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.030699 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 03:46:02.031075 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.020574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37151 (local address 127.30.24.1:55525)
0806 03:46:02.020781 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.020821 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.021381 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.021669 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.021672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.022364 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:02.022598 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.022603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.023373 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.023376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.024508 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.024512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.024908 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.024909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.025068 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.025677 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:02.025686 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:02.027047 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:02.029079 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:02.029087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:02.029090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:02.029414 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:02.029690 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:02.029692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:02.029692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:02.029852 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:02.030216 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:02.030225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:02.030421 (+ 196us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.030569 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.030791 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260806 03:46:02.031754 15978 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37151
I20260806 03:46:02.031878 15978 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:02.032102 15978 heartbeater.cc:511] Master 127.30.24.62:37151 requested a full tablet report, sending...
I20260806 03:46:02.032723 15780 ts_manager.cc:194] Registered new tserver with Master: d428c26828ae4ded957277e30a80eea9 (127.30.24.1:40145)
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
I20260806 03:46:02.033545 15780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:55525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:46:02.051715 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37151
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:02.135859 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:02.136075 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:02.136098 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:02.137326 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:02.137370 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:02.137398 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:46:02.138866 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:02.139138 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:02.139384 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:02.144661 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.144716 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:02.144734 15983 server_base.cc:1034] running on GCE node
W20260806 03:46:02.144697 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:02.145167 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:02.145387 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:02.146534 15983 hybrid_clock.cc:648] HybridClock initialized: now 1785987962146513 us; error 38 us; skew 500 ppm
Aug 06 03:46:02 dist-test-slave-10pc krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:02.149336 15983 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:46:02.149843 15983 webserver.cc:460] Webserver started at http://127.30.24.2:38281/ using document root <none> and password file <none>
I20260806 03:46:02.150053 15983 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:02.150094 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:02.150187 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:02.151226 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "153bffe5b9cf43e89977560358e935d5"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "4af3d0a697547237f6d49e12c2cd11be"
server_key_iv: "26aa1597c4af8ef73833d317a227ab56"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.151543 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "153bffe5b9cf43e89977560358e935d5"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "4af3d0a697547237f6d49e12c2cd11be"
server_key_iv: "26aa1597c4af8ef73833d317a227ab56"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.152899 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260806 03:46:02.153638 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.153826 15983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:02.153901 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "153bffe5b9cf43e89977560358e935d5"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "4af3d0a697547237f6d49e12c2cd11be"
server_key_iv: "26aa1597c4af8ef73833d317a227ab56"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.153957 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:02.177244 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:02.179535 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:02.179657 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:02.179852 15983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:02.180191 15983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:02.180228 15983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.180251 15983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:02.180297 15983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.186328 15983 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:37793
I20260806 03:46:02.186373 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:37793 every 8 connection(s)
I20260806 03:46:02.186712 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:46:02.187975 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 15983
I20260806 03:46:02.188066 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:46:02.188269 30816 external_mini_cluster.cc:1428] Setting key 60d9fa8cbd7e581ddcfeb438e8e73b94
Aug 06 03:46:02 dist-test-slave-10pc krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:02.197774 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.187480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:57137 (local address 127.30.24.62:37151)
0806 03:46:02.187540 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.187542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.187790 (+ 248us) server_negotiation.cc:407] Connection header received
0806 03:46:02.188476 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.188477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.188491 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.188546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:02.189673 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.190259 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.191717 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.191930 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.191992 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.194730 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:02.194731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:02.194732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:02.194739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:02.196189 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.196748 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.196749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.196749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.196802 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.197113 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.197114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.197114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.197221 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:02.197272 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.197626 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.197706 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 03:46:02.198089 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.187471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37151 (local address 127.30.24.2:57137)
0806 03:46:02.187662 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.187700 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.188352 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.188626 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.188631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.189314 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:02.189545 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.189551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.190392 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.190393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.191618 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.191622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.192020 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.192021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.192202 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.192828 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:02.192837 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:02.194631 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:02.196276 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:02.196285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:02.196288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:02.196673 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:02.196876 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:02.196877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:02.196877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:02.197040 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:02.197289 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:02.197296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:02.197477 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.197611 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.197805 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260806 03:46:02.198702 16113 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37151
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
I20260806 03:46:02.198827 16113 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:02.199049 16113 heartbeater.cc:511] Master 127.30.24.62:37151 requested a full tablet report, sending...
I20260806 03:46:02.199436 15780 ts_manager.cc:194] Registered new tserver with Master: 153bffe5b9cf43e89977560358e935d5 (127.30.24.2:37793)
I20260806 03:46:02.199879 15780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:57137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:46:02.217523 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37151
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:02.310410 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:02.310638 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:02.310664 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:02.311837 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:02.311887 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:02.311915 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:46:02.313531 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:37151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:02.313797 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:02.314049 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:02.319296 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.319284 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.319284 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:02.319612 16118 server_base.cc:1034] running on GCE node
I20260806 03:46:02.319833 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:02.320087 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:02.321257 16118 hybrid_clock.cc:648] HybridClock initialized: now 1785987962321241 us; error 29 us; skew 500 ppm
Aug 06 03:46:02 dist-test-slave-10pc krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:02.323797 16118 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:46:02.324258 16118 webserver.cc:460] Webserver started at http://127.30.24.3:45099/ using document root <none> and password file <none>
I20260806 03:46:02.324550 16118 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:02.324621 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:02.324790 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:02.325922 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "cde3a96064bc4bc0af17d9ba89d2b161"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "6cbcf1bcee871fc2ab79257b45bb4247"
server_key_iv: "2c1762622c5805a4b71f22ade50dd21e"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.326313 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "cde3a96064bc4bc0af17d9ba89d2b161"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "6cbcf1bcee871fc2ab79257b45bb4247"
server_key_iv: "2c1762622c5805a4b71f22ade50dd21e"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.327682 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:02.328548 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.328733 16118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:02.328835 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "cde3a96064bc4bc0af17d9ba89d2b161"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "6cbcf1bcee871fc2ab79257b45bb4247"
server_key_iv: "2c1762622c5805a4b71f22ade50dd21e"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.328924 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:02.338749 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:02.341320 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:02.341491 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:02.341755 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:02.342113 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:02.342177 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.342221 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:02.342248 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.348239 16118 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:39017
I20260806 03:46:02.348312 16248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:39017 every 8 connection(s)
I20260806 03:46:02.348716 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:46:02.354110 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16118
I20260806 03:46:02.354215 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:46:02.354426 30816 external_mini_cluster.cc:1428] Setting key 4696db96c4ad35e881530f516f91686d
Aug 06 03:46:02 dist-test-slave-10pc krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:02.359198 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.349550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:54279 (local address 127.30.24.62:37151)
0806 03:46:02.349608 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.349610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.349894 (+ 284us) server_negotiation.cc:407] Connection header received
0806 03:46:02.350420 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.350421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.350433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.350490 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:02.351492 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.352003 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.353139 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.353340 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.353403 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.356050 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:02.356052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:02.356052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:02.356062 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:02.357681 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.358143 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.358145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.358145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.358219 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.358483 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.358484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.358484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.358615 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:02.358706 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.359001 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.359086 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 03:46:02.359403 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.349507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:37151 (local address 127.30.24.3:54279)
0806 03:46:02.349716 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.349784 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.350329 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.350581 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.350585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.351188 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:02.351397 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.351402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.352104 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.352105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.353058 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.353060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.353442 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.353443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.353556 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.354108 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:02.354121 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:02.355960 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:02.357806 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:02.357815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:02.357818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:02.358062 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:02.358292 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:02.358292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:02.358293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:02.358414 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:02.358695 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:02.358702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:02.358859 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.358978 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.359154 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260806 03:46:02.359961 16249 heartbeater.cc:347] Connected to a master server at 127.30.24.62:37151
I20260806 03:46:02.360056 16249 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:02.360217 16249 heartbeater.cc:511] Master 127.30.24.62:37151 requested a full tablet report, sending...
I20260806 03:46:02.360608 15780 ts_manager.cc:194] Registered new tserver with Master: cde3a96064bc4bc0af17d9ba89d2b161 (127.30.24.3:39017)
I20260806 03:46:02.361001 15780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:54279
I20260806 03:46:02.366487 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:46:02.374212 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.367989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.30.24.62:37151)
0806 03:46:02.368060 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.368063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.368197 (+ 134us) server_negotiation.cc:407] Connection header received
0806 03:46:02.368269 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.368270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.368279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.368330 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:02.368960 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.369469 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.370165 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.370351 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.370405 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.371582 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:02.371583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:02.371583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:02.371590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:02.373047 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.373414 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.373415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.373415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.373468 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:02.373682 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:02.373683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:02.373683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:02.373828 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:02.373905 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.373994 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.374056 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:46:02.376000 15780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:46:02.377782 15780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:46:02.384555 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.380242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:37793 (local address 127.0.0.1:60824)
0806 03:46:02.380583 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.380633 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.380715 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.381183 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.381184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.381187 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:02.381235 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.381236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.382516 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.382517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.383498 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.383501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.384166 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.384167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.384255 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.384270 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.384375 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.384482 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":35,"thread_start_us":111,"threads_started":1}
I20260806 03:46:02.384568 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.380358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.30.24.2:37793)
0806 03:46:02.380567 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.380577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.380713 (+ 136us) server_negotiation.cc:407] Connection header received
0806 03:46:02.381040 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.381043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.381058 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.381176 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:02.381289 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.382403 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.383581 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.384074 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.384164 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.384226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.384398 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.384446 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20260806 03:46:02.384663 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.380359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.30.24.1:40145)
0806 03:46:02.380656 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.380660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.380690 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:46:02.380824 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.380827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.380843 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.380918 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:02.381131 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.381949 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.383330 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.384079 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.384149 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.384212 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.384517 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.384562 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260806 03:46:02.384866 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.380776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:39017 (local address 127.0.0.1:59608)
0806 03:46:02.380917 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.380927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.381004 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.381275 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.381276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.381278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:02.381321 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.381323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.382569 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.382570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.383598 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.383601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.384610 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.384611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.384676 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.384680 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.384742 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.384784 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260806 03:46:02.384948 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.380600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.30.24.3:39017)
0806 03:46:02.380870 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.380875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.380978 (+ 103us) server_negotiation.cc:407] Connection header received
0806 03:46:02.381096 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.381099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.381116 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.381171 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:02.381404 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.382484 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.383704 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.384421 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.384517 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.384574 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.384786 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.384845 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260806 03:46:02.384866 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.380241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:40145 (local address 127.0.0.1:39542)
0806 03:46:02.380583 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.380632 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.380715 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.380944 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.380947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.380949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:02.381028 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.381030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.382064 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.382064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.383236 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.383239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.384250 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.384251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.384316 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.384320 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.384370 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.384786 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":81,"threads_started":1}
I20260806 03:46:02.385870 16047 tablet_service.cc:1467] Processing CreateTablet for tablet ab9a5ec5fe2f4626b5dde92a68563af0 (DEFAULT_TABLE table=test-table [id=7e102be39cf344bcbfcb0d06817ebbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:02.385890 15909 tablet_service.cc:1467] Processing CreateTablet for tablet ab9a5ec5fe2f4626b5dde92a68563af0 (DEFAULT_TABLE table=test-table [id=7e102be39cf344bcbfcb0d06817ebbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:02.385886 16182 tablet_service.cc:1467] Processing CreateTablet for tablet ab9a5ec5fe2f4626b5dde92a68563af0 (DEFAULT_TABLE table=test-table [id=7e102be39cf344bcbfcb0d06817ebbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:02.386248 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9a5ec5fe2f4626b5dde92a68563af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:02.386248 15909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9a5ec5fe2f4626b5dde92a68563af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:02.386248 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9a5ec5fe2f4626b5dde92a68563af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:02.389123 16266 tablet_bootstrap.cc:492] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Bootstrap starting.
I20260806 03:46:02.389792 16268 tablet_bootstrap.cc:492] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: Bootstrap starting.
I20260806 03:46:02.389954 16266 tablet_bootstrap.cc:654] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:02.390528 16268 tablet_bootstrap.cc:654] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:02.390604 16266 log.cc:824] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:02.391160 16268 log.cc:824] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:02.391489 16267 tablet_bootstrap.cc:492] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Bootstrap starting.
I20260806 03:46:02.391580 16266 tablet_bootstrap.cc:492] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: No bootstrap required, opened a new log
I20260806 03:46:02.391665 16266 ts_tablet_manager.cc:1397] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:46:02.391959 16268 tablet_bootstrap.cc:492] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: No bootstrap required, opened a new log
I20260806 03:46:02.392045 16268 ts_tablet_manager.cc:1397] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:46:02.392264 16267 tablet_bootstrap.cc:654] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:02.392884 16267 log.cc:824] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:02.393734 16267 tablet_bootstrap.cc:492] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: No bootstrap required, opened a new log
I20260806 03:46:02.393810 16267 ts_tablet_manager.cc:1397] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260806 03:46:02.393947 16268 raft_consensus.cc:348] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.393931 16266 raft_consensus.cc:348] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.394124 16268 raft_consensus.cc:374] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:02.394129 16266 raft_consensus.cc:374] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:02.394168 16266 raft_consensus.cc:729] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153bffe5b9cf43e89977560358e935d5, State: Initialized, Role: FOLLOWER
I20260806 03:46:02.394188 16268 raft_consensus.cc:729] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d428c26828ae4ded957277e30a80eea9, State: Initialized, Role: FOLLOWER
I20260806 03:46:02.394310 16268 consensus_queue.cc:260] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.394337 16266 consensus_queue.cc:260] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.394613 16266 ts_tablet_manager.cc:1428] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:46:02.394640 16268 ts_tablet_manager.cc:1428] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:46:02.394641 15978 heartbeater.cc:503] Master 127.30.24.62:37151 was elected leader, sending a full tablet report...
I20260806 03:46:02.394778 16113 heartbeater.cc:503] Master 127.30.24.62:37151 was elected leader, sending a full tablet report...
I20260806 03:46:02.396064 16267 raft_consensus.cc:348] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.396255 16267 raft_consensus.cc:374] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:02.396284 16267 raft_consensus.cc:729] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde3a96064bc4bc0af17d9ba89d2b161, State: Initialized, Role: FOLLOWER
I20260806 03:46:02.396420 16267 consensus_queue.cc:260] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.396780 16249 heartbeater.cc:503] Master 127.30.24.62:37151 was elected leader, sending a full tablet report...
I20260806 03:46:02.396777 16267 ts_tablet_manager.cc:1428] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260806 03:46:02.437294 16114 tablet.cc:2369] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:46:02.520516 15979 tablet.cc:2369] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:02.576179 16275 raft_consensus.cc:482] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:46:02.576327 16275 raft_consensus.cc:504] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.576809 16275 leader_election.cc:288] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d428c26828ae4ded957277e30a80eea9 (127.30.24.1:40145), 153bffe5b9cf43e89977560358e935d5 (127.30.24.2:37793)
I20260806 03:46:02.580046 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.577215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:40627 (local address 127.30.24.2:37793)
0806 03:46:02.577283 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.577285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.577298 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:46:02.577334 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.577335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.577349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.577417 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:02.577627 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.578180 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.579186 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.579653 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.579750 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.579761 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.579928 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.579968 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:02.580055 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.577275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:55409 (local address 127.30.24.1:40145)
0806 03:46:02.577362 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.577364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.577372 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:02.577439 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.577440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.577452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.577512 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:02.577734 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.578192 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.579292 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.579644 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.579700 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.579709 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.579937 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.579977 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 03:46:02.580129 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.577081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:37793 (local address 127.30.24.3:40627)
0806 03:46:02.577151 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.577165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.577211 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.577440 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.577441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.577443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:02.577534 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.577537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.578280 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.578281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.579086 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.579088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.579779 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.579780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.579844 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.579847 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.579918 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.579959 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":47}
I20260806 03:46:02.580224 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.577103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:40145 (local address 127.30.24.3:55409)
0806 03:46:02.577287 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.577299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.577357 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.577547 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.577548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.577549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:02.577649 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.577650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.578312 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.578313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.579199 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.579202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.579749 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.579750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.579840 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.579845 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.579917 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.579963 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260806 03:46:02.580526 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a5ec5fe2f4626b5dde92a68563af0" candidate_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153bffe5b9cf43e89977560358e935d5" is_pre_election: true
I20260806 03:46:02.580724 16067 raft_consensus.cc:2454] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde3a96064bc4bc0af17d9ba89d2b161 in term 0.
I20260806 03:46:02.580720 15929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a5ec5fe2f4626b5dde92a68563af0" candidate_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d428c26828ae4ded957277e30a80eea9" is_pre_election: true
I20260806 03:46:02.580891 15929 raft_consensus.cc:2454] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde3a96064bc4bc0af17d9ba89d2b161 in term 0.
I20260806 03:46:02.580977 16137 leader_election.cc:302] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153bffe5b9cf43e89977560358e935d5, cde3a96064bc4bc0af17d9ba89d2b161; no voters:
I20260806 03:46:02.581140 16275 raft_consensus.cc:2781] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:46:02.581200 16275 raft_consensus.cc:482] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:46:02.581226 16275 raft_consensus.cc:3037] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:02.582010 16275 raft_consensus.cc:504] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.582182 16275 leader_election.cc:288] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [CANDIDATE]: Term 1 election: Requested vote from peers d428c26828ae4ded957277e30a80eea9 (127.30.24.1:40145), 153bffe5b9cf43e89977560358e935d5 (127.30.24.2:37793)
I20260806 03:46:02.582340 15929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a5ec5fe2f4626b5dde92a68563af0" candidate_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d428c26828ae4ded957277e30a80eea9"
I20260806 03:46:02.582379 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a5ec5fe2f4626b5dde92a68563af0" candidate_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153bffe5b9cf43e89977560358e935d5"
I20260806 03:46:02.582436 15929 raft_consensus.cc:3037] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:02.582469 16067 raft_consensus.cc:3037] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:02.583230 15929 raft_consensus.cc:2454] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde3a96064bc4bc0af17d9ba89d2b161 in term 1.
I20260806 03:46:02.583240 16067 raft_consensus.cc:2454] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde3a96064bc4bc0af17d9ba89d2b161 in term 1.
I20260806 03:46:02.583439 16136 leader_election.cc:302] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cde3a96064bc4bc0af17d9ba89d2b161, d428c26828ae4ded957277e30a80eea9; no voters:
I20260806 03:46:02.583575 16275 raft_consensus.cc:2781] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:02.583794 16275 raft_consensus.cc:686] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 1 LEADER]: Becoming Leader. State: Replica: cde3a96064bc4bc0af17d9ba89d2b161, State: Running, Role: LEADER
I20260806 03:46:02.583915 16275 consensus_queue.cc:237] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } }
I20260806 03:46:02.585084 15779 catalog_manager.cc:5495] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 reported cstate change: term changed from 0 to 1, leader changed from <none> to cde3a96064bc4bc0af17d9ba89d2b161 (127.30.24.3). New cstate: current_term: 1 leader_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde3a96064bc4bc0af17d9ba89d2b161" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 39017 } health_report { overall_health: HEALTHY } } }
W20260806 03:46:02.599639 16250 tablet.cc:2369] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:02.615635 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.613205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.30.24.3:39017)
0806 03:46:02.613263 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.613265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.613275 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:46:02.613323 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.613324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.613349 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.613429 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:02.613559 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.614027 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.614803 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.615003 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.615076 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.615275 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:02.615379 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:02.615478 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.615498 (+ 20us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.615519 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:46:02.617457 16067 raft_consensus.cc:1261] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 1 FOLLOWER]: Refusing update from remote peer cde3a96064bc4bc0af17d9ba89d2b161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:02.617457 15929 raft_consensus.cc:1261] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 1 FOLLOWER]: Refusing update from remote peer cde3a96064bc4bc0af17d9ba89d2b161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:02.617856 16277 consensus_queue.cc:1035] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d428c26828ae4ded957277e30a80eea9" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:02.618014 16275 consensus_queue.cc:1035] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153bffe5b9cf43e89977560358e935d5" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:02.622593 15778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:02.622701 15778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:02.623530 15778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1586edbd4c4f72a2332342c14e62a3: Sending DeleteTablet for 3 replicas of tablet ab9a5ec5fe2f4626b5dde92a68563af0
I20260806 03:46:02.623888 15909 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9a5ec5fe2f4626b5dde92a68563af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:02 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:39542
I20260806 03:46:02.623894 16047 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9a5ec5fe2f4626b5dde92a68563af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:02 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:60824
I20260806 03:46:02.624346 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 15845
I20260806 03:46:02.624375 16290 tablet_replica.cc:330] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9: stopping tablet replica
I20260806 03:46:02.624496 16290 raft_consensus.cc:2229] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:02.624476 16182 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9a5ec5fe2f4626b5dde92a68563af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:02 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:59608
I20260806 03:46:02.624609 16290 raft_consensus.cc:2258] T ab9a5ec5fe2f4626b5dde92a68563af0 P d428c26828ae4ded957277e30a80eea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:02.624763 16291 tablet_replica.cc:330] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: stopping tablet replica
I20260806 03:46:02.624847 16291 raft_consensus.cc:2229] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:02.624934 16291 raft_consensus.cc:2258] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:02.630102 16291 ts_tablet_manager.cc:1905] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:02.631619 16291 ts_tablet_manager.cc:1918] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:02.631717 16291 log.cc:1196] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/ab9a5ec5fe2f4626b5dde92a68563af0
I20260806 03:46:02.632043 16291 ts_tablet_manager.cc:1939] T ab9a5ec5fe2f4626b5dde92a68563af0 P 153bffe5b9cf43e89977560358e935d5: Deleting consensus metadata
I20260806 03:46:02.632520 15766 catalog_manager.cc:4841] TS 153bffe5b9cf43e89977560358e935d5 (127.30.24.2:37793): tablet ab9a5ec5fe2f4626b5dde92a68563af0 (table test-table [id=7e102be39cf344bcbfcb0d06817ebbf3]) successfully deleted
W20260806 03:46:02.634454 15765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:40145 (error 108)
I20260806 03:46:02.635020 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.634869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:40145 (local address 127.0.0.1:39546)
0806 03:46:02.634929 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.634970 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 03:46:02.635092 16292 tablet_replica.cc:330] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: stopping tablet replica
I20260806 03:46:02.635149 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 15983
I20260806 03:46:02.635217 16292 raft_consensus.cc:2229] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:46:02.635324 16292 raft_consensus.cc:2258] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 03:46:02.637737 15765 catalog_manager.cc:4584] TS d428c26828ae4ded957277e30a80eea9 (127.30.24.1:40145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab9a5ec5fe2f4626b5dde92a68563af0: Network error: Client connection negotiation failed: client connection to 127.30.24.1:40145: connect: Connection refused (error 111)
I20260806 03:46:02.637903 16292 ts_tablet_manager.cc:1905] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:02.639613 16292 ts_tablet_manager.cc:1918] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:02.639736 16292 log.cc:1196] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/ab9a5ec5fe2f4626b5dde92a68563af0
I20260806 03:46:02.640029 16292 ts_tablet_manager.cc:1939] T ab9a5ec5fe2f4626b5dde92a68563af0 P cde3a96064bc4bc0af17d9ba89d2b161: Deleting consensus metadata
I20260806 03:46:02.640548 15766 catalog_manager.cc:4841] TS cde3a96064bc4bc0af17d9ba89d2b161 (127.30.24.3:39017): tablet ab9a5ec5fe2f4626b5dde92a68563af0 (table test-table [id=7e102be39cf344bcbfcb0d06817ebbf3]) successfully deleted
I20260806 03:46:02.642508 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16118
I20260806 03:46:02.647917 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 15748
2026-08-06T03:46:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2120 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T03:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:02Z Disabled control of system clock
I20260806 03:46:02.677538 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:41109
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:02.760178 16297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:02.760412 16297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:02.760433 16297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:02.761591 16297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:02.761636 16297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:02.761649 16297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:02.761663 16297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:02.761677 16297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:02.762966 16297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:41109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:02.763185 16297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:02.763432 16297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:02.768617 16305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.768638 16303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.768642 16302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:02.769002 16297 server_base.cc:1034] running on GCE node
I20260806 03:46:02.769266 16297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:02.769587 16297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:02.770756 16297 hybrid_clock.cc:648] HybridClock initialized: now 1785987962770751 us; error 45 us; skew 500 ppm
I20260806 03:46:02.772157 16297 webserver.cc:460] Webserver started at http://127.30.24.62:35609/ using document root <none> and password file <none>
I20260806 03:46:02.772426 16297 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:02.772548 16297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:02.772711 16297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:02.773985 16297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "8ed67c69439d4337b8c5a2df3b64523e"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "da15e715367719c75d629034cb514220"
server_key_iv: "d2f3e0aed2bf69bd7143dbec648999f7"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.774371 16297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "8ed67c69439d4337b8c5a2df3b64523e"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "da15e715367719c75d629034cb514220"
server_key_iv: "d2f3e0aed2bf69bd7143dbec648999f7"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.775717 16297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:02.776633 16311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.776826 16297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:02.776940 16297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "8ed67c69439d4337b8c5a2df3b64523e"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "da15e715367719c75d629034cb514220"
server_key_iv: "d2f3e0aed2bf69bd7143dbec648999f7"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.777077 16297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:02.810907 16297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:02.811321 16297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:02.811510 16297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:02.816063 16297 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:41109
I20260806 03:46:02.816110 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:41109 every 8 connection(s)
I20260806 03:46:02.816524 16297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:02.817539 16365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:02.819875 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e: Bootstrap starting.
I20260806 03:46:02.820504 16365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:02.820825 16365 log.cc:824] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:02.821609 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e: No bootstrap required, opened a new log
I20260806 03:46:02.822386 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16297
I20260806 03:46:02.822520 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:02.822723 30816 external_mini_cluster.cc:1428] Setting key f03fcd3f1c5d33ed7748ba1ee17b680a
I20260806 03:46:02.823925 16365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed67c69439d4337b8c5a2df3b64523e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 41109 } }
I20260806 03:46:02.824103 16365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:02.824141 16365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed67c69439d4337b8c5a2df3b64523e, State: Initialized, Role: FOLLOWER
I20260806 03:46:02.824311 16365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed67c69439d4337b8c5a2df3b64523e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 41109 } }
I20260806 03:46:02.824404 16365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:02.824437 16365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:02.824527 16365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:02.825395 16365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed67c69439d4337b8c5a2df3b64523e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 41109 } }
I20260806 03:46:02.825511 16365 leader_election.cc:302] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed67c69439d4337b8c5a2df3b64523e; no voters:
I20260806 03:46:02.825682 16365 leader_election.cc:288] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:02.825767 16370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:02.825943 16365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:02.825979 16370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [term 1 LEADER]: Becoming Leader. State: Replica: 8ed67c69439d4337b8c5a2df3b64523e, State: Running, Role: LEADER
I20260806 03:46:02.826177 16370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed67c69439d4337b8c5a2df3b64523e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 41109 } }
I20260806 03:46:02.826660 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed67c69439d4337b8c5a2df3b64523e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed67c69439d4337b8c5a2df3b64523e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 41109 } } }
I20260806 03:46:02.826764 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:02.827345 16379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:02.827883 16379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:02.828644 16372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed67c69439d4337b8c5a2df3b64523e. Latest consensus state: current_term: 1 leader_uuid: "8ed67c69439d4337b8c5a2df3b64523e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed67c69439d4337b8c5a2df3b64523e" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 41109 } } }
I20260806 03:46:02.828747 16372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:02.829835 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.823290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.30.24.62:41109)
0806 03:46:02.823535 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.823545 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.823583 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:46:02.823882 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.823891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.823972 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.824132 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:02.824366 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.826243 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.827538 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.828126 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.828266 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.829055 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:02.829058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:02.829058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:02.829263 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:02.829319 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:02.829335 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:02.829403 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.829533 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.829615 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260806 03:46:02.832847 16379 catalog_manager.cc:1347] Generated new cluster ID: ff13157ca78f465e838e89f0271b5e3c
I20260806 03:46:02.832903 16379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:02.838932 16379 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:02.839430 16379 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:02.842852 16379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e: Generated new TSK 0
I20260806 03:46:02.843020 16379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:02.847440 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:41109
--builtin_ntp_servers=127.30.24.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:02.934698 16389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:02.935052 16389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:02.935117 16389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:02.936614 16389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:02.936666 16389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:02.936693 16389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:46:02.938076 16389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:41109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:02.938336 16389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:02.938647 16389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:02.943892 16395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.943893 16394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:02.943913 16397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:02.944617 16389 server_base.cc:1034] running on GCE node
I20260806 03:46:02.944850 16389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:02.945109 16389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:02.946296 16389 hybrid_clock.cc:648] HybridClock initialized: now 1785987962946292 us; error 46 us; skew 500 ppm
I20260806 03:46:02.947700 16389 webserver.cc:460] Webserver started at http://127.30.24.1:35585/ using document root <none> and password file <none>
I20260806 03:46:02.947980 16389 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:02.948051 16389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:02.948225 16389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:02.949398 16389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "8c8362f8030a46a484a0ccd15695320f"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "52b0e496e01f7d93dd5c37c48f535703"
server_key_iv: "b10640a5a36079dc2bb04b1eb703b78f"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.949767 16389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "8c8362f8030a46a484a0ccd15695320f"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "52b0e496e01f7d93dd5c37c48f535703"
server_key_iv: "b10640a5a36079dc2bb04b1eb703b78f"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.951217 16389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:46:02.952108 16403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.952332 16389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:02.952441 16389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "8c8362f8030a46a484a0ccd15695320f"
format_stamp: "Formatted at 2026-08-06 03:46:02 on dist-test-slave-10pc"
server_key: "52b0e496e01f7d93dd5c37c48f535703"
server_key_iv: "b10640a5a36079dc2bb04b1eb703b78f"
server_key_version: "encryptionkey@0"
I20260806 03:46:02.952566 16389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:02.973412 16389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:02.973767 16389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:02.974027 16389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:02.974305 16389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:02.974673 16389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:02.974709 16389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.974761 16389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:02.974786 16389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:02.981321 16389 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:43727
I20260806 03:46:02.981403 16517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:43727 every 8 connection(s)
I20260806 03:46:02.981772 16389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:46:02.986117 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16389
I20260806 03:46:02.986296 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:46:02.986543 30816 external_mini_cluster.cc:1428] Setting key 789acebcca3557b9f7761deea5797d29
I20260806 03:46:02.987977 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.982623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:43163 (local address 127.30.24.62:41109)
0806 03:46:02.982693 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:02.982695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:02.982946 (+ 251us) server_negotiation.cc:407] Connection header received
0806 03:46:02.983526 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:02.983528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:02.983543 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:02.983608 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:02.984387 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.984971 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.986004 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.986256 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.986320 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.987516 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:02.987517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:02.987518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:02.987524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:02.987540 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:02.987544 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:02.987618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:02.987858 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 03:46:02.987893 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:02.988022 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:41109
--builtin_ntp_servers=127.30.24.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:46:02.988209 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:02.982637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:41109 (local address 127.30.24.1:43163)
0806 03:46:02.982813 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:02.982850 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:02.983408 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:02.983690 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:02.983693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:02.983703 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:02.984265 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.984269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.985076 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.985077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.985896 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:02.985898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:02.986406 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:02.986408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:02.986609 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:02.987360 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:02.987361 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:02.987394 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:02.987421 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:02.987757 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:02.987764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:02.987767 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:02.987847 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:46:02.987966 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260806 03:46:02.988680 16518 heartbeater.cc:347] Connected to a master server at 127.30.24.62:41109
I20260806 03:46:02.988808 16518 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:02.988998 16518 heartbeater.cc:511] Master 127.30.24.62:41109 requested a full tablet report, sending...
I20260806 03:46:02.989558 16329 ts_manager.cc:194] Registered new tserver with Master: 8c8362f8030a46a484a0ccd15695320f (127.30.24.1:43727)
I20260806 03:46:02.990267 16329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.1:43163
W20260806 03:46:03.073119 16521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:03.073335 16521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:03.073356 16521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:03.074559 16521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:03.074602 16521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:03.074628 16521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:46:03.075992 16521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:41109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:03.076205 16521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:03.076547 16521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:03.081851 16527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:03.081849 16528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:03.082067 16521 server_base.cc:1034] running on GCE node
W20260806 03:46:03.081849 16530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:03.082386 16521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:03.082664 16521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:03.083835 16521 hybrid_clock.cc:648] HybridClock initialized: now 1785987963083830 us; error 38 us; skew 500 ppm
I20260806 03:46:03.085204 16521 webserver.cc:460] Webserver started at http://127.30.24.2:39983/ using document root <none> and password file <none>
I20260806 03:46:03.085477 16521 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:03.085562 16521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:03.085709 16521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:03.086820 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "d06b4e4f092d40e290bc489e70d5a209"
format_stamp: "Formatted at 2026-08-06 03:46:03 on dist-test-slave-10pc"
server_key: "bd6588168f031445e91844851b247d53"
server_key_iv: "48e6ab8be1748bd355ee07c586b02306"
server_key_version: "encryptionkey@0"
I20260806 03:46:03.087209 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "d06b4e4f092d40e290bc489e70d5a209"
format_stamp: "Formatted at 2026-08-06 03:46:03 on dist-test-slave-10pc"
server_key: "bd6588168f031445e91844851b247d53"
server_key_iv: "48e6ab8be1748bd355ee07c586b02306"
server_key_version: "encryptionkey@0"
I20260806 03:46:03.088666 16521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:46:03.089490 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:03.089709 16521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:03.089812 16521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "d06b4e4f092d40e290bc489e70d5a209"
format_stamp: "Formatted at 2026-08-06 03:46:03 on dist-test-slave-10pc"
server_key: "bd6588168f031445e91844851b247d53"
server_key_iv: "48e6ab8be1748bd355ee07c586b02306"
server_key_version: "encryptionkey@0"
I20260806 03:46:03.089886 16521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:03.111554 16521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:03.111912 16521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:03.112072 16521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:03.112326 16521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:03.112717 16521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:03.112771 16521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:03.112804 16521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:03.112845 16521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:03.119103 16521 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:35421
I20260806 03:46:03.119155 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:35421 every 8 connection(s)
I20260806 03:46:03.119504 16521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:46:03.124317 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16521
I20260806 03:46:03.124459 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:46:03.124691 30816 external_mini_cluster.cc:1428] Setting key 974fa23ca5293e6fc3326eaf310e5779
I20260806 03:46:03.125424 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.120301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48159 (local address 127.30.24.62:41109)
0806 03:46:03.120381 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.120384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.120705 (+ 321us) server_negotiation.cc:407] Connection header received
0806 03:46:03.121199 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.121201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.121212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.121279 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:03.122009 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.122537 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.123691 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.123891 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.123958 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.125102 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:03.125103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:03.125104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:03.125109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:03.125127 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:03.125131 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:03.125205 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.125308 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.125343 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 03:46:03.125717 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.120267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:41109 (local address 127.30.24.2:48159)
0806 03:46:03.120500 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.120543 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.121043 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.121362 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.121365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.121373 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:03.121909 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.121913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.122646 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.122654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.123562 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.123566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.123981 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.123982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.124182 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.124931 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:03.124931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:03.124962 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:03.124987 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:03.125206 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:03.125213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:03.125216 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.125310 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.125425 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260806 03:46:03.125860 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:41109
--builtin_ntp_servers=127.30.24.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:46:03.126394 16651 heartbeater.cc:347] Connected to a master server at 127.30.24.62:41109
I20260806 03:46:03.126536 16651 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:03.126760 16651 heartbeater.cc:511] Master 127.30.24.62:41109 requested a full tablet report, sending...
I20260806 03:46:03.127193 16329 ts_manager.cc:194] Registered new tserver with Master: d06b4e4f092d40e290bc489e70d5a209 (127.30.24.2:35421)
I20260806 03:46:03.127624 16329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.2:48159
W20260806 03:46:03.213568 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:03.213780 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:03.213802 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:03.214965 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:03.215009 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:03.215036 16654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:46:03.216404 16654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:41109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:03.216676 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:03.216995 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:03.222106 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:03.222271 16654 server_base.cc:1034] running on GCE node
W20260806 03:46:03.222080 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:03.222096 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:03.222577 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:03.222806 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:03.223977 16654 hybrid_clock.cc:648] HybridClock initialized: now 1785987963223970 us; error 35 us; skew 500 ppm
I20260806 03:46:03.225641 16654 webserver.cc:460] Webserver started at http://127.30.24.3:38583/ using document root <none> and password file <none>
I20260806 03:46:03.225975 16654 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:03.226055 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:03.226228 16654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:03.227716 16654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "14c892916111444593313bdd7983179f"
format_stamp: "Formatted at 2026-08-06 03:46:03 on dist-test-slave-10pc"
server_key: "59b1856a20b31c90078b40dad3f26ab8"
server_key_iv: "377967f04fed1634706d4058204eb31c"
server_key_version: "encryptionkey@0"
I20260806 03:46:03.228245 16654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "14c892916111444593313bdd7983179f"
format_stamp: "Formatted at 2026-08-06 03:46:03 on dist-test-slave-10pc"
server_key: "59b1856a20b31c90078b40dad3f26ab8"
server_key_iv: "377967f04fed1634706d4058204eb31c"
server_key_version: "encryptionkey@0"
I20260806 03:46:03.229627 16654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:46:03.230414 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:03.230583 16654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:46:03.230657 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "14c892916111444593313bdd7983179f"
format_stamp: "Formatted at 2026-08-06 03:46:03 on dist-test-slave-10pc"
server_key: "59b1856a20b31c90078b40dad3f26ab8"
server_key_iv: "377967f04fed1634706d4058204eb31c"
server_key_version: "encryptionkey@0"
I20260806 03:46:03.230711 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:03.265235 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:03.265542 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:03.265650 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:03.265861 16654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:03.266156 16654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:03.266187 16654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:03.266211 16654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:03.266223 16654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:03.271917 16654 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:40885
I20260806 03:46:03.272008 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:40885 every 8 connection(s)
I20260806 03:46:03.272395 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:46:03.277782 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.273270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:33333 (local address 127.30.24.62:41109)
0806 03:46:03.273341 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.273344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.273629 (+ 285us) server_negotiation.cc:407] Connection header received
0806 03:46:03.274118 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.274119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.274130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.274179 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:03.274960 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.275464 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.276473 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.276677 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.276739 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.277548 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:03.277549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:03.277549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:03.277554 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:03.277569 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:03.277573 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:03.277625 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.277694 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.277722 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 03:46:03.277953 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.273271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:41109 (local address 127.30.24.3:33333)
0806 03:46:03.273491 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.273540 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.274033 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.274239 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.274243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.274251 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:03.274868 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.274874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.275565 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.275566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.276349 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.276351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.276749 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.276750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.276864 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.277409 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:03.277410 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:03.277440 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:03.277464 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:03.277628 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:03.277630 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:03.277632 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.277678 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.277757 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260806 03:46:03.278528 16785 heartbeater.cc:347] Connected to a master server at 127.30.24.62:41109
I20260806 03:46:03.278650 16785 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:03.278841 16785 heartbeater.cc:511] Master 127.30.24.62:41109 requested a full tablet report, sending...
I20260806 03:46:03.279251 16329 ts_manager.cc:194] Registered new tserver with Master: 14c892916111444593313bdd7983179f (127.30.24.3:40885)
I20260806 03:46:03.279624 16329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.3:33333
I20260806 03:46:03.282584 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16654
I20260806 03:46:03.282711 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:46:03.282910 30816 external_mini_cluster.cc:1428] Setting key 739baf400a9936ba2da16af0f9d84092
I20260806 03:46:03.283897 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:46:03.288434 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.285315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.30.24.62:41109)
0806 03:46:03.285377 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.285379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.285566 (+ 187us) server_negotiation.cc:407] Connection header received
0806 03:46:03.285623 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.285624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.285639 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.285688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:03.285838 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.286330 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.287147 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.287337 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.287410 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.288113 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:03.288114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:03.288115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:03.288119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:03.288134 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:03.288138 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:03.288196 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.288295 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.288319 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:46:03.290042 16329 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:46:03.291594 16329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:46:03.298084 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.294237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.30.24.1:43727)
0806 03:46:03.294420 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.294426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.294439 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:46:03.294522 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.294525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.294541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.294831 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:03.294895 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.295646 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.297021 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.297658 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.297718 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.297786 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.297938 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.297999 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260806 03:46:03.298076 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.294027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:43727 (local address 127.0.0.1:39458)
0806 03:46:03.294317 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.294363 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.294430 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.294653 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.294656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.294660 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:03.294736 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.294738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.295736 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.295737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.296925 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.296928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.297742 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.297743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.297823 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.297838 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.297921 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.297994 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":87,"threads_started":1}
I20260806 03:46:03.298759 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.294031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:35421 (local address 127.0.0.1:59940)
0806 03:46:03.294555 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.294565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.294601 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.294918 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.294919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.294921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:03.294956 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.294957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.296222 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.296223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.297216 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.297219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.298049 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.298051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.298117 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.298122 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.298629 (+ 507us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.298666 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260806 03:46:03.299114 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.294272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.30.24.2:35421)
0806 03:46:03.294708 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.294714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.294724 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:46:03.294761 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.294763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.294779 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.294850 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:03.295052 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.296133 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.297321 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.297919 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.297984 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.298067 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.298953 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.299018 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":105,"threads_started":1}
I20260806 03:46:03.299086 16452 tablet_service.cc:1467] Processing CreateTablet for tablet b4b89ba3871c40c09599093e18b01aa4 (DEFAULT_TABLE table=test-table [id=eb6d98d010e04eff820ae58cb04efda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:03.299296 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.294028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40885 (local address 127.0.0.1:52108)
0806 03:46:03.294406 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.294418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.294474 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.294769 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.294771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.294773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:03.294831 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.294833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.296205 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.296206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.297232 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.297235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.298731 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.298732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.299106 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.299111 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.299168 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.299218 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260806 03:46:03.299418 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.294236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.30.24.3:40885)
0806 03:46:03.294538 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.294542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.294551 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:46:03.294594 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.294597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.294669 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.294914 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:03.294958 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.296106 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.298106 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.298640 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.298716 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.298797 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.299291 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.299336 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":72,"threads_started":1}
I20260806 03:46:03.299487 16452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b89ba3871c40c09599093e18b01aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:03.300074 16716 tablet_service.cc:1467] Processing CreateTablet for tablet b4b89ba3871c40c09599093e18b01aa4 (DEFAULT_TABLE table=test-table [id=eb6d98d010e04eff820ae58cb04efda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:03.300391 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b89ba3871c40c09599093e18b01aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:03.301205 16585 tablet_service.cc:1467] Processing CreateTablet for tablet b4b89ba3871c40c09599093e18b01aa4 (DEFAULT_TABLE table=test-table [id=eb6d98d010e04eff820ae58cb04efda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:03.301501 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b89ba3871c40c09599093e18b01aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:03.303097 16800 tablet_bootstrap.cc:492] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f: Bootstrap starting.
I20260806 03:46:03.303653 16801 tablet_bootstrap.cc:492] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Bootstrap starting.
I20260806 03:46:03.303826 16800 tablet_bootstrap.cc:654] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:03.304183 16800 log.cc:824] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:03.304317 16801 tablet_bootstrap.cc:654] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:03.304708 16801 log.cc:824] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:03.304698 16803 tablet_bootstrap.cc:492] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Bootstrap starting.
I20260806 03:46:03.305400 16800 tablet_bootstrap.cc:492] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f: No bootstrap required, opened a new log
I20260806 03:46:03.305462 16801 tablet_bootstrap.cc:492] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: No bootstrap required, opened a new log
I20260806 03:46:03.305456 16803 tablet_bootstrap.cc:654] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:03.305502 16800 ts_tablet_manager.cc:1397] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:46:03.305517 16801 ts_tablet_manager.cc:1397] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 03:46:03.305783 16803 log.cc:824] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:03.306617 16803 tablet_bootstrap.cc:492] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: No bootstrap required, opened a new log
I20260806 03:46:03.306684 16803 ts_tablet_manager.cc:1397] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 03:46:03.307595 16800 raft_consensus.cc:348] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.307760 16800 raft_consensus.cc:374] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:03.307803 16800 raft_consensus.cc:729] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8362f8030a46a484a0ccd15695320f, State: Initialized, Role: FOLLOWER
I20260806 03:46:03.307919 16800 consensus_queue.cc:260] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.308144 16801 raft_consensus.cc:348] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.308233 16518 heartbeater.cc:503] Master 127.30.24.62:41109 was elected leader, sending a full tablet report...
I20260806 03:46:03.308296 16801 raft_consensus.cc:374] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:03.308341 16801 raft_consensus.cc:729] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c892916111444593313bdd7983179f, State: Initialized, Role: FOLLOWER
I20260806 03:46:03.308220 16800 ts_tablet_manager.cc:1428] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:46:03.308517 16801 consensus_queue.cc:260] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.308760 16801 ts_tablet_manager.cc:1428] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 03:46:03.308878 16785 heartbeater.cc:503] Master 127.30.24.62:41109 was elected leader, sending a full tablet report...
I20260806 03:46:03.309226 16803 raft_consensus.cc:348] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.309422 16803 raft_consensus.cc:374] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:03.309474 16803 raft_consensus.cc:729] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06b4e4f092d40e290bc489e70d5a209, State: Initialized, Role: FOLLOWER
I20260806 03:46:03.309619 16803 consensus_queue.cc:260] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.309893 16803 ts_tablet_manager.cc:1428] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 03:46:03.309958 16651 heartbeater.cc:503] Master 127.30.24.62:41109 was elected leader, sending a full tablet report...
W20260806 03:46:03.370059 16652 tablet.cc:2369] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:03.419209 16808 raft_consensus.cc:482] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:46:03.419389 16808 raft_consensus.cc:504] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.419842 16808 leader_election.cc:288] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c892916111444593313bdd7983179f (127.30.24.3:40885), 8c8362f8030a46a484a0ccd15695320f (127.30.24.1:43727)
I20260806 03:46:03.423718 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.420193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:37343 (local address 127.30.24.1:43727)
0806 03:46:03.420243 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.420244 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.420484 (+ 240us) server_negotiation.cc:407] Connection header received
0806 03:46:03.420517 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.420518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.420541 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.420635 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:03.420823 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.421497 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.422800 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.423371 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.423435 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.423444 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.423606 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.423644 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 03:46:03.423718 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.420172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:48819 (local address 127.30.24.3:40885)
0806 03:46:03.420221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.420224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.420263 (+ 39us) server_negotiation.cc:407] Connection header received
0806 03:46:03.420296 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.420297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.420315 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.420357 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:03.420598 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.421332 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.422770 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.423328 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.423408 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.423421 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.423606 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.423647 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:46:03.423738 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.420043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:40885 (local address 127.30.24.2:48819)
0806 03:46:03.420145 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.420161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.420212 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.420391 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.420392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.420395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:03.420507 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.420518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.421419 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.421420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.422642 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.422646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.423433 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.423434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.423523 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.423526 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.423580 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.423641 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260806 03:46:03.423748 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.420077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:43727 (local address 127.30.24.2:37343)
0806 03:46:03.420323 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.420331 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:03.420408 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:03.420674 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:03.420675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:03.420678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:03.420737 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.420739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.421581 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.421582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.422697 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:03.422699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.423448 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.423448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:03.423514 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.423519 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:03.423580 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 03:46:03.423641 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":83,"thread_start_us":101,"threads_started":1}
I20260806 03:46:03.424228 16472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b89ba3871c40c09599093e18b01aa4" candidate_uuid: "d06b4e4f092d40e290bc489e70d5a209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8362f8030a46a484a0ccd15695320f" is_pre_election: true
I20260806 03:46:03.424430 16472 raft_consensus.cc:2454] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d06b4e4f092d40e290bc489e70d5a209 in term 0.
I20260806 03:46:03.424425 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b89ba3871c40c09599093e18b01aa4" candidate_uuid: "d06b4e4f092d40e290bc489e70d5a209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c892916111444593313bdd7983179f" is_pre_election: true
I20260806 03:46:03.424636 16737 raft_consensus.cc:2454] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d06b4e4f092d40e290bc489e70d5a209 in term 0.
I20260806 03:46:03.424711 16540 leader_election.cc:302] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c8362f8030a46a484a0ccd15695320f, d06b4e4f092d40e290bc489e70d5a209; no voters:
I20260806 03:46:03.424862 16808 raft_consensus.cc:2781] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:46:03.424944 16808 raft_consensus.cc:482] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:46:03.424962 16808 raft_consensus.cc:3037] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:03.425839 16808 raft_consensus.cc:504] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.426085 16808 leader_election.cc:288] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [CANDIDATE]: Term 1 election: Requested vote from peers 14c892916111444593313bdd7983179f (127.30.24.3:40885), 8c8362f8030a46a484a0ccd15695320f (127.30.24.1:43727)
I20260806 03:46:03.426142 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b89ba3871c40c09599093e18b01aa4" candidate_uuid: "d06b4e4f092d40e290bc489e70d5a209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c892916111444593313bdd7983179f"
I20260806 03:46:03.426239 16737 raft_consensus.cc:3037] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:03.426288 16472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b89ba3871c40c09599093e18b01aa4" candidate_uuid: "d06b4e4f092d40e290bc489e70d5a209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8362f8030a46a484a0ccd15695320f"
I20260806 03:46:03.426379 16472 raft_consensus.cc:3037] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:03.427217 16737 raft_consensus.cc:2454] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d06b4e4f092d40e290bc489e70d5a209 in term 1.
I20260806 03:46:03.427274 16472 raft_consensus.cc:2454] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d06b4e4f092d40e290bc489e70d5a209 in term 1.
I20260806 03:46:03.427440 16539 leader_election.cc:302] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c892916111444593313bdd7983179f, d06b4e4f092d40e290bc489e70d5a209; no voters:
I20260806 03:46:03.427549 16808 raft_consensus.cc:2781] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:03.427702 16808 raft_consensus.cc:686] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 1 LEADER]: Becoming Leader. State: Replica: d06b4e4f092d40e290bc489e70d5a209, State: Running, Role: LEADER
I20260806 03:46:03.427800 16808 consensus_queue.cc:237] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } }
I20260806 03:46:03.428784 16329 catalog_manager.cc:5495] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 reported cstate change: term changed from 0 to 1, leader changed from <none> to d06b4e4f092d40e290bc489e70d5a209 (127.30.24.2). New cstate: current_term: 1 leader_uuid: "d06b4e4f092d40e290bc489e70d5a209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d06b4e4f092d40e290bc489e70d5a209" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 35421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 } health_report { overall_health: UNKNOWN } } }
I20260806 03:46:03.443588 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.441153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.30.24.2:35421)
0806 03:46:03.441217 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:03.441220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:03.441228 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:03.441270 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:03.441270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:03.441289 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:03.441367 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:03.441479 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.441974 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.442750 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:03.442945 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:03.443024 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:03.443227 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:03.443332 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:03.443429 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:03.443448 (+ 19us) server_negotiation.cc:299] Negotiation successful
0806 03:46:03.443468 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:03.445492 16472 raft_consensus.cc:1261] T b4b89ba3871c40c09599093e18b01aa4 P 8c8362f8030a46a484a0ccd15695320f [term 1 FOLLOWER]: Refusing update from remote peer d06b4e4f092d40e290bc489e70d5a209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:03.445492 16737 raft_consensus.cc:1261] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 1 FOLLOWER]: Refusing update from remote peer d06b4e4f092d40e290bc489e70d5a209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:03.445881 16808 consensus_queue.cc:1035] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c892916111444593313bdd7983179f" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:03.446031 16810 consensus_queue.cc:1035] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c8362f8030a46a484a0ccd15695320f" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 43727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:03.448467 16813 mvcc.cc:205] Tried to move back new op lower bound from 7315406698269958144 to 7315406698200915968. Current Snapshot: MvccSnapshot[applied={T|T < 7315406698269958144}]
I20260806 03:46:03.451227 16329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:03.451308 16329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:03.452071 16329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ed67c69439d4337b8c5a2df3b64523e: Sending DeleteTablet for 3 replicas of tablet b4b89ba3871c40c09599093e18b01aa4
I20260806 03:46:03.452427 16716 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b89ba3871c40c09599093e18b01aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:03 UTC) from {username='slave'} at 127.0.0.1:52108
I20260806 03:46:03.452430 16585 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b89ba3871c40c09599093e18b01aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:03 UTC) from {username='slave'} at 127.0.0.1:59940
I20260806 03:46:03.452567 16452 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b89ba3871c40c09599093e18b01aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:03 UTC) from {username='slave'} at 127.0.0.1:39458
I20260806 03:46:03.452701 16822 tablet_replica.cc:330] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: stopping tablet replica
I20260806 03:46:03.452804 16822 raft_consensus.cc:2229] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:46:03.452926 16822 raft_consensus.cc:2258] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:03.452934 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16389
I20260806 03:46:03.454684 16823 tablet_replica.cc:330] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: stopping tablet replica
I20260806 03:46:03.454797 16823 raft_consensus.cc:2229] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:03.454908 16823 raft_consensus.cc:2258] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:03.455369 16823 ts_tablet_manager.cc:1905] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:03.456833 16822 ts_tablet_manager.cc:1905] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:03.457369 16823 ts_tablet_manager.cc:1918] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:03.457451 16823 log.cc:1196] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/b4b89ba3871c40c09599093e18b01aa4
I20260806 03:46:03.457717 16823 ts_tablet_manager.cc:1939] T b4b89ba3871c40c09599093e18b01aa4 P 14c892916111444593313bdd7983179f: Deleting consensus metadata
I20260806 03:46:03.458163 16314 catalog_manager.cc:4841] TS 14c892916111444593313bdd7983179f (127.30.24.3:40885): tablet b4b89ba3871c40c09599093e18b01aa4 (table test-table [id=eb6d98d010e04eff820ae58cb04efda1]) successfully deleted
I20260806 03:46:03.458998 16822 ts_tablet_manager.cc:1918] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:03.459133 16822 log.cc:1196] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/b4b89ba3871c40c09599093e18b01aa4
I20260806 03:46:03.459461 16822 ts_tablet_manager.cc:1939] T b4b89ba3871c40c09599093e18b01aa4 P d06b4e4f092d40e290bc489e70d5a209: Deleting consensus metadata
I20260806 03:46:03.459913 16313 catalog_manager.cc:4841] TS d06b4e4f092d40e290bc489e70d5a209 (127.30.24.2:35421): tablet b4b89ba3871c40c09599093e18b01aa4 (table test-table [id=eb6d98d010e04eff820ae58cb04efda1]) successfully deleted
W20260806 03:46:03.461467 16315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:43727 (error 108)
I20260806 03:46:03.461808 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16521
I20260806 03:46:03.462704 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:03.461871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:43727 (local address 127.0.0.1:39464)
0806 03:46:03.462508 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:03.462651 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613}
W20260806 03:46:03.463441 16315 catalog_manager.cc:4584] TS 8c8362f8030a46a484a0ccd15695320f (127.30.24.1:43727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4b89ba3871c40c09599093e18b01aa4: Network error: Client connection negotiation failed: client connection to 127.30.24.1:43727: connect: Connection refused (error 111)
I20260806 03:46:03.468978 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16654
I20260806 03:46:03.473987 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16297
2026-08-06T03:46:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (825 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:03 dist-test-slave-10pc krb5kdc[16828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:03 dist-test-slave-10pc krb5kdc[16828](info): set up 2 sockets
Aug 06 03:46:03 dist-test-slave-10pc krb5kdc[16828](info): commencing operation
krb5kdc: starting...
W20260806 03:46:04.605598 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:04 dist-test-slave-10pc krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:04.731927 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42539
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:04.817230 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:04.817476 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:04.817512 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:04.818635 16841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:04.818701 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:04.818727 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:04.818759 16841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:04.818784 16841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:04.820255 16841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:42539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:04.820554 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:04.820791 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:04.825625 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:04.825634 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:04.825644 16849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:04.825860 16841 server_base.cc:1034] running on GCE node
I20260806 03:46:04.826088 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:04.826388 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:04.827584 16841 hybrid_clock.cc:648] HybridClock initialized: now 1785987964827574 us; error 36 us; skew 500 ppm
Aug 06 03:46:04 dist-test-slave-10pc krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:04.834440 16841 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:04.834959 16841 webserver.cc:460] Webserver started at http://127.30.24.62:45505/ using document root <none> and password file <none>
I20260806 03:46:04.835209 16841 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:04.835281 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:04.835433 16841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:04.836640 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "b2212335658c4bcbbf540f807c0f38aa"
format_stamp: "Formatted at 2026-08-06 03:46:04 on dist-test-slave-10pc"
server_key: "cfcd5953fab3dd170c2f19462135aa76"
server_key_iv: "4ab0cd6d053954be562fbe40540c2e67"
server_key_version: "encryptionkey@0"
I20260806 03:46:04.837028 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "b2212335658c4bcbbf540f807c0f38aa"
format_stamp: "Formatted at 2026-08-06 03:46:04 on dist-test-slave-10pc"
server_key: "cfcd5953fab3dd170c2f19462135aa76"
server_key_iv: "4ab0cd6d053954be562fbe40540c2e67"
server_key_version: "encryptionkey@0"
I20260806 03:46:04.838414 16841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:46:04.839143 16856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:04.839318 16841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:46:04.839426 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "b2212335658c4bcbbf540f807c0f38aa"
format_stamp: "Formatted at 2026-08-06 03:46:04 on dist-test-slave-10pc"
server_key: "cfcd5953fab3dd170c2f19462135aa76"
server_key_iv: "4ab0cd6d053954be562fbe40540c2e67"
server_key_version: "encryptionkey@0"
I20260806 03:46:04.839501 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:04.856381 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:04.858397 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:04.858567 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:04.862792 16841 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:42539
I20260806 03:46:04.862836 16909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:42539 every 8 connection(s)
I20260806 03:46:04.863236 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:04.864096 16910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:04.866532 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: Bootstrap starting.
I20260806 03:46:04.867143 16910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:04.867441 16910 log.cc:824] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:04.867931 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16841
I20260806 03:46:04.868072 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:04.868335 30816 external_mini_cluster.cc:1428] Setting key e5e77379d099f73d2605336c0b1f805c
I20260806 03:46:04.868353 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: No bootstrap required, opened a new log
I20260806 03:46:04.870839 16910 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2212335658c4bcbbf540f807c0f38aa" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42539 } }
I20260806 03:46:04.870977 16910 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:04.871007 16910 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2212335658c4bcbbf540f807c0f38aa, State: Initialized, Role: FOLLOWER
I20260806 03:46:04.871151 16910 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2212335658c4bcbbf540f807c0f38aa" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42539 } }
I20260806 03:46:04.871223 16910 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:04.871264 16910 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:04.871315 16910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:04.871922 16910 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2212335658c4bcbbf540f807c0f38aa" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42539 } }
I20260806 03:46:04.872045 16910 leader_election.cc:302] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2212335658c4bcbbf540f807c0f38aa; no voters:
I20260806 03:46:04.872248 16910 leader_election.cc:288] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:04.872273 16915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:04.872406 16915 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [term 1 LEADER]: Becoming Leader. State: Replica: b2212335658c4bcbbf540f807c0f38aa, State: Running, Role: LEADER
I20260806 03:46:04.872578 16910 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:04.872597 16915 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2212335658c4bcbbf540f807c0f38aa" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42539 } }
I20260806 03:46:04.873014 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2212335658c4bcbbf540f807c0f38aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2212335658c4bcbbf540f807c0f38aa" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42539 } } }
I20260806 03:46:04.873023 16916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2212335658c4bcbbf540f807c0f38aa. Latest consensus state: current_term: 1 leader_uuid: "b2212335658c4bcbbf540f807c0f38aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2212335658c4bcbbf540f807c0f38aa" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 42539 } } }
I20260806 03:46:04.873119 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:04.873132 16916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa [sys.catalog]: This master's current role is: LEADER
W20260806 03:46:04.874498 16931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:04.874564 16931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:04.874615 16922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:04.874817 16922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 03:46:04 dist-test-slave-10pc krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:04.876327 16922 catalog_manager.cc:1347] Generated new cluster ID: 1edab8b756f64002b2a7ac5228da8fce
I20260806 03:46:04.876379 16922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:04.879961 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:04.868940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.30.24.62:42539)
0806 03:46:04.869149 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:04.869156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:04.869182 (+ 26us) server_negotiation.cc:407] Connection header received
0806 03:46:04.869746 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:04.869755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:04.869804 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:04.869918 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:04.870743 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:04.872090 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:04.873209 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:04.873841 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:04.873951 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:04.876754 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:04.876758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:04.876758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:04.876771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:04.878469 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:04.878821 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:04.878823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:04.878824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:04.878914 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:04.879127 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:04.879128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:04.879128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:04.879380 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:04.879528 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:04.879642 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 03:46:04.879802 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260806 03:46:04.898792 16922 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:04.899401 16922 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:04.902015 16922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: Generated new TSK 0
I20260806 03:46:04.902223 16922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:46:04.936345 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42539
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:05.022222 16936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:05.022450 16936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:05.022475 16936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:05.023662 16936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:05.023708 16936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:05.023736 16936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:46:05.025279 16936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:05.025535 16936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:05.025794 16936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:05.031095 16942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:05.031177 16936 server_base.cc:1034] running on GCE node
W20260806 03:46:05.031095 16944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.031129 16941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:05.031514 16936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:05.031773 16936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:05.032950 16936 hybrid_clock.cc:648] HybridClock initialized: now 1785987965032937 us; error 67 us; skew 500 ppm
Aug 06 03:46:05 dist-test-slave-10pc krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:05.035542 16936 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:46:05.036002 16936 webserver.cc:460] Webserver started at http://127.30.24.1:41139/ using document root <none> and password file <none>
I20260806 03:46:05.036219 16936 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:05.036262 16936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:05.036356 16936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:05.037462 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "befd2ed674dd4459997a525e4d574032"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "e5e4257cce7afe134a84434130472375"
server_key_iv: "697f0bb17f2cf390f6dee6e0152da8de"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.037776 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "befd2ed674dd4459997a525e4d574032"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "e5e4257cce7afe134a84434130472375"
server_key_iv: "697f0bb17f2cf390f6dee6e0152da8de"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.039072 16936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:05.039819 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.039994 16936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:05.040086 16936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "befd2ed674dd4459997a525e4d574032"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "e5e4257cce7afe134a84434130472375"
server_key_iv: "697f0bb17f2cf390f6dee6e0152da8de"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.040174 16936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:05.052785 16936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:05.054657 16936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:05.054826 16936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:05.055068 16936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:05.055392 16936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:05.055447 16936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.055491 16936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:05.055516 16936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.061479 16936 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:36363
I20260806 03:46:05.061525 17066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:36363 every 8 connection(s)
I20260806 03:46:05.061848 16936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:46:05.062441 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 16936
I20260806 03:46:05.062561 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:46:05.062804 30816 external_mini_cluster.cc:1428] Setting key cfce0f56e450d43960ae696b1a6d095f
Aug 06 03:46:05 dist-test-slave-10pc krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
I20260806 03:46:05.072134 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.062622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:41997 (local address 127.30.24.62:42539)
0806 03:46:05.062690 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.062692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.062894 (+ 202us) server_negotiation.cc:407] Connection header received
0806 03:46:05.063494 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.063495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.063508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.063554 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:05.064612 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.065131 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.066164 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.066397 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.066482 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.069147 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:05.069148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:05.069148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:05.069155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:05.070608 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.071115 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.071116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.071116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.071168 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.071458 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.071458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.071459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.071561 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:05.071609 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.071982 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.072064 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:46:05.072384 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.062590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42539 (local address 127.30.24.1:41997)
0806 03:46:05.062771 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.062814 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.063388 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.063626 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.063630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.064260 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:05.064508 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.064514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.065252 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.065253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.066057 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.066060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.066496 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.066497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.066631 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.067174 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:05.067183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:05.069045 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:05.070706 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:05.070715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:05.070717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:05.071032 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:05.071238 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:05.071239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:05.071239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:05.071387 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:05.071627 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:05.071635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:05.071831 (+ 196us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.071958 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.072135 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260806 03:46:05.073007 17067 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42539
I20260806 03:46:05.073117 17067 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:05.073330 17067 heartbeater.cc:511] Master 127.30.24.62:42539 requested a full tablet report, sending...
I20260806 03:46:05.073971 16873 ts_manager.cc:194] Registered new tserver with Master: befd2ed674dd4459997a525e4d574032 (127.30.24.1:36363)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
I20260806 03:46:05.074828 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:41997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:46:05.091439 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42539
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:05.176254 17072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:05.176527 17072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:05.176573 17072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:05.177739 17072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:05.177781 17072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:05.177807 17072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:46:05.179318 17072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:05.179525 17072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:05.179713 17072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:46:05.185029 17072 server_base.cc:1034] running on GCE node
W20260806 03:46:05.185063 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.185143 17080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.185057 17078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:05.185459 17072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:05.185724 17072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:05.186892 17072 hybrid_clock.cc:648] HybridClock initialized: now 1785987965186877 us; error 47 us; skew 500 ppm
Aug 06 03:46:05 dist-test-slave-10pc krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:05.189515 17072 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:46:05.189973 17072 webserver.cc:460] Webserver started at http://127.30.24.2:32887/ using document root <none> and password file <none>
I20260806 03:46:05.190238 17072 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:05.190310 17072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:05.190460 17072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:05.191550 17072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "ee194878204548d49a9c5dd84041e585"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "f1f2b9573461b30fedbfdd7be9406b2d"
server_key_iv: "8986988fb2748edbdc8e1ab6ffc0b4bb"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.191932 17072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "ee194878204548d49a9c5dd84041e585"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "f1f2b9573461b30fedbfdd7be9406b2d"
server_key_iv: "8986988fb2748edbdc8e1ab6ffc0b4bb"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.193378 17072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:46:05.194118 17087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.194291 17072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:05.194396 17072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "ee194878204548d49a9c5dd84041e585"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "f1f2b9573461b30fedbfdd7be9406b2d"
server_key_iv: "8986988fb2748edbdc8e1ab6ffc0b4bb"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.194479 17072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:05.212013 17072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:05.214429 17072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:05.214601 17072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:05.214850 17072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:05.215183 17072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:05.215236 17072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.215277 17072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:05.215306 17072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.221591 17072 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:38319
I20260806 03:46:05.221674 17202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:38319 every 8 connection(s)
I20260806 03:46:05.222011 17072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:46:05.227427 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17072
I20260806 03:46:05.227567 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:46:05.227816 30816 external_mini_cluster.cc:1428] Setting key dbd8937d1e4b9925c795f751c36a4107
Aug 06 03:46:05 dist-test-slave-10pc krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:05.232148 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.222817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:59981 (local address 127.30.24.62:42539)
0806 03:46:05.222869 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.222872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.223099 (+ 227us) server_negotiation.cc:407] Connection header received
0806 03:46:05.223579 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.223580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.223591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.223645 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:05.224654 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.225196 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.226169 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.226351 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.226404 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.228948 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:05.228949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:05.228950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:05.228957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:05.230641 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.231149 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.231150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.231151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.231203 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.231494 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.231494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.231494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.231609 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:05.231660 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.232015 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.232085 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:46:05.232515 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.222806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42539 (local address 127.30.24.2:59981)
0806 03:46:05.222974 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.223012 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.223495 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.223708 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.223711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.224289 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:05.224565 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.224570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.225285 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.225286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.226091 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.226093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.226424 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.226424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.226535 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.227047 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:05.227056 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:05.228851 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:05.230732 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:05.230740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:05.230743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:05.231069 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:05.231271 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:05.231272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:05.231272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:05.231427 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:05.231675 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:05.231682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:05.231866 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.231994 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.232188 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260806 03:46:05.233004 17203 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42539
I20260806 03:46:05.233107 17203 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:05.233294 17203 heartbeater.cc:511] Master 127.30.24.62:42539 requested a full tablet report, sending...
I20260806 03:46:05.234104 16873 ts_manager.cc:194] Registered new tserver with Master: ee194878204548d49a9c5dd84041e585 (127.30.24.2:38319)
I20260806 03:46:05.234663 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:59981
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:46:05.258989 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42539
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:05.345595 17208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:05.345814 17208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:05.345839 17208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:05.346998 17208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:05.347043 17208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:05.347097 17208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:46:05.348683 17208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:05.348933 17208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:05.349185 17208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:05.354681 17213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.354694 17214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.354663 17216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:05.354928 17208 server_base.cc:1034] running on GCE node
I20260806 03:46:05.355181 17208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:05.355387 17208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:05.356534 17208 hybrid_clock.cc:648] HybridClock initialized: now 1785987965356506 us; error 40 us; skew 500 ppm
Aug 06 03:46:05 dist-test-slave-10pc krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:05.359277 17208 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:46:05.359822 17208 webserver.cc:460] Webserver started at http://127.30.24.3:41721/ using document root <none> and password file <none>
I20260806 03:46:05.360059 17208 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:05.360105 17208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:05.360286 17208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:05.361487 17208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "b9e2ddfb87f34a7da6216e68a02e14dc"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "cff9aacb467639215591148f11a22923"
server_key_iv: "b8767a8a5e6777ee04b46e66298eaaa0"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.361866 17208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "b9e2ddfb87f34a7da6216e68a02e14dc"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "cff9aacb467639215591148f11a22923"
server_key_iv: "b8767a8a5e6777ee04b46e66298eaaa0"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.363281 17208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:05.364143 17223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.364336 17208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:05.364444 17208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "b9e2ddfb87f34a7da6216e68a02e14dc"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "cff9aacb467639215591148f11a22923"
server_key_iv: "b8767a8a5e6777ee04b46e66298eaaa0"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.364543 17208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:05.391229 17208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:05.393796 17208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:05.393980 17208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:05.394266 17208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:05.394610 17208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:05.394673 17208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.394742 17208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:05.394773 17208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.400929 17208 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:46825
I20260806 03:46:05.400988 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:46825 every 8 connection(s)
I20260806 03:46:05.401360 17208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:46:05.405897 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17208
I20260806 03:46:05.406013 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:46:05.406252 30816 external_mini_cluster.cc:1428] Setting key e5d380e16c5c130b7fbb3ea53b880309
Aug 06 03:46:05 dist-test-slave-10pc krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:05.411140 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.402133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:37183 (local address 127.30.24.62:42539)
0806 03:46:05.402192 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.402194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.402459 (+ 265us) server_negotiation.cc:407] Connection header received
0806 03:46:05.402994 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.402995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.403006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.403058 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:05.403982 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.404540 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.405541 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.405743 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.405809 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.408395 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:05.408397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:05.408397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:05.408406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:05.409720 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.410129 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.410130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.410130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.410196 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.410456 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.410457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.410457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.410581 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:05.410655 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.410964 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.411045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 03:46:05.411381 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.402130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:42539 (local address 127.30.24.3:37183)
0806 03:46:05.402313 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.402351 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.402899 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.403115 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.403118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.403704 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:05.403903 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.403908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.404618 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.404619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.405427 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.405429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.405837 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.405839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.405993 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.406682 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:05.406693 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:05.408294 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:05.409805 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:05.409813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:05.409816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:05.410049 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:05.410265 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:05.410266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:05.410266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:05.410379 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:05.410661 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:05.410669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:05.410822 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.410928 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.411105 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260806 03:46:05.411881 17338 heartbeater.cc:347] Connected to a master server at 127.30.24.62:42539
I20260806 03:46:05.411993 17338 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:05.412189 17338 heartbeater.cc:511] Master 127.30.24.62:42539 requested a full tablet report, sending...
I20260806 03:46:05.412643 16873 ts_manager.cc:194] Registered new tserver with Master: b9e2ddfb87f34a7da6216e68a02e14dc (127.30.24.3:46825)
I20260806 03:46:05.413038 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:37183
I20260806 03:46:05.418139 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:46:05.425738 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.419570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.30.24.62:42539)
0806 03:46:05.419636 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.419637 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.419665 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:46:05.419736 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.419737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.419745 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.419788 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:05.420363 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.420920 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.421638 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.421822 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.421872 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.423237 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:05.423238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:05.423238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:05.423245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:05.424529 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.424937 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.424938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.424938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.424990 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:05.425233 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:05.425235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:05.425235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:05.425394 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:05.425491 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.425609 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.425672 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 03:46:05.427460 16873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:46:05.429207 16873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:46:05.436172 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.431881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.30.24.1:36363)
0806 03:46:05.432077 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.432081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.432363 (+ 282us) server_negotiation.cc:407] Connection header received
0806 03:46:05.432574 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.432577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.432592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.432676 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:05.432875 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.433557 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435112 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.435831 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435894 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.435948 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.436045 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.436082 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260806 03:46:05.436172 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.431613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:36363 (local address 127.0.0.1:54868)
0806 03:46:05.432221 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.432230 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.432401 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.432752 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.432752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.432754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:05.432797 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.432798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.433665 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.433666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.435029 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.435032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.435897 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.435967 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.435972 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.436029 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.436073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":59,"thread_start_us":101,"threads_started":1}
I20260806 03:46:05.436368 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.431868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.30.24.3:46825)
0806 03:46:05.432150 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.432153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.432167 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:46:05.432212 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.432214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.432226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.432318 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:05.432564 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.433672 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435287 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.435866 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435935 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.435996 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.436219 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.436262 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260806 03:46:05.436400 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.431607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:38319 (local address 127.0.0.1:36114)
0806 03:46:05.431790 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.431832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.431900 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.432284 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.432287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.432290 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:05.432376 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.432378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.433788 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.433789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.434845 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.434849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.435794 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.435879 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.435895 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.435961 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.436021 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":101,"thread_start_us":77,"threads_started":1}
I20260806 03:46:05.436414 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.431728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.30.24.2:38319)
0806 03:46:05.432096 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.432101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.432112 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:05.432158 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.432160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.432177 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.432264 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:05.432493 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.433670 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.434942 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.435680 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.435760 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.435825 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.436123 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.436180 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":89,"threads_started":1}
I20260806 03:46:05.436601 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.431611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:46825 (local address 127.0.0.1:39214)
0806 03:46:05.432055 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.432067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.432135 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.432398 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.432399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.432401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:05.432438 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.432439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.433867 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.433868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.435200 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.435202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.436030 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.436031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.436089 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.436093 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.436143 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.436180 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":29,"thread_start_us":113,"threads_started":1}
I20260806 03:46:05.437383 17136 tablet_service.cc:1467] Processing CreateTablet for tablet 283fc7919d3e4b05980369a44adf1dd8 (DEFAULT_TABLE table=test-table [id=40ec796e10e946feb728fb1b1e61a787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:05.437398 17272 tablet_service.cc:1467] Processing CreateTablet for tablet 283fc7919d3e4b05980369a44adf1dd8 (DEFAULT_TABLE table=test-table [id=40ec796e10e946feb728fb1b1e61a787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:05.437388 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 283fc7919d3e4b05980369a44adf1dd8 (DEFAULT_TABLE table=test-table [id=40ec796e10e946feb728fb1b1e61a787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:05.437707 17136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 283fc7919d3e4b05980369a44adf1dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:05.437740 17272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 283fc7919d3e4b05980369a44adf1dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:05.437740 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 283fc7919d3e4b05980369a44adf1dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:05.440613 17355 tablet_bootstrap.cc:492] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Bootstrap starting.
I20260806 03:46:05.441108 17356 tablet_bootstrap.cc:492] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Bootstrap starting.
I20260806 03:46:05.441344 17357 tablet_bootstrap.cc:492] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: Bootstrap starting.
I20260806 03:46:05.441512 17355 tablet_bootstrap.cc:654] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:05.441929 17356 tablet_bootstrap.cc:654] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:05.442001 17357 tablet_bootstrap.cc:654] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:05.442185 17355 log.cc:824] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:05.442555 17356 log.cc:824] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:05.442684 17357 log.cc:824] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:05.443177 17355 tablet_bootstrap.cc:492] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: No bootstrap required, opened a new log
I20260806 03:46:05.443256 17355 ts_tablet_manager.cc:1397] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:46:05.443574 17356 tablet_bootstrap.cc:492] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: No bootstrap required, opened a new log
I20260806 03:46:05.443682 17356 ts_tablet_manager.cc:1397] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:46:05.443729 17357 tablet_bootstrap.cc:492] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: No bootstrap required, opened a new log
I20260806 03:46:05.443799 17357 ts_tablet_manager.cc:1397] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:46:05.445227 17356 raft_consensus.cc:348] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.445405 17356 raft_consensus.cc:374] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:05.445451 17356 raft_consensus.cc:729] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e2ddfb87f34a7da6216e68a02e14dc, State: Initialized, Role: FOLLOWER
I20260806 03:46:05.445401 17355 raft_consensus.cc:348] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.445592 17355 raft_consensus.cc:374] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:05.445640 17355 raft_consensus.cc:729] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee194878204548d49a9c5dd84041e585, State: Initialized, Role: FOLLOWER
I20260806 03:46:05.445602 17356 consensus_queue.cc:260] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.445756 17355 consensus_queue.cc:260] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.445910 17356 ts_tablet_manager.cc:1428] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 03:46:05.445963 17338 heartbeater.cc:503] Master 127.30.24.62:42539 was elected leader, sending a full tablet report...
I20260806 03:46:05.445847 17357 raft_consensus.cc:348] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.446025 17357 raft_consensus.cc:374] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:05.446041 17355 ts_tablet_manager.cc:1428] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:46:05.446065 17357 raft_consensus.cc:729] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: befd2ed674dd4459997a525e4d574032, State: Initialized, Role: FOLLOWER
I20260806 03:46:05.446189 17203 heartbeater.cc:503] Master 127.30.24.62:42539 was elected leader, sending a full tablet report...
I20260806 03:46:05.446173 17357 consensus_queue.cc:260] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.446842 17357 ts_tablet_manager.cc:1428] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 03:46:05.446969 17067 heartbeater.cc:503] Master 127.30.24.62:42539 was elected leader, sending a full tablet report...
W20260806 03:46:05.472708 17204 tablet.cc:2369] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:46:05.562641 17068 tablet.cc:2369] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:05.620378 17361 raft_consensus.cc:482] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:46:05.620532 17361 raft_consensus.cc:504] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.620927 17361 leader_election.cc:288] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee194878204548d49a9c5dd84041e585 (127.30.24.2:38319), befd2ed674dd4459997a525e4d574032 (127.30.24.1:36363)
I20260806 03:46:05.624982 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.621280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:43821 (local address 127.30.24.2:38319)
0806 03:46:05.621336 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.621339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.621351 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:46:05.621389 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.621390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.621405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.621486 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:05.621648 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.622485 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.623913 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.624610 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.624679 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.624691 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.624856 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.624899 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:46:05.624989 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.621139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:38319 (local address 127.30.24.3:43821)
0806 03:46:05.621221 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.621234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.621289 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.621485 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.621487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.621489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:05.621571 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.621573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.622579 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.622579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.623796 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.623799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.624697 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.624698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.624774 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.624776 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.624832 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.624899 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":23}
I20260806 03:46:05.624995 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.621259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:36363 (local address 127.30.24.3:34681)
0806 03:46:05.621480 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.621490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:05.621549 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:05.621720 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:05.621721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:05.621722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:05.621772 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.621774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.622736 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.622737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.623811 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:05.623813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.624688 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.624690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:05.624766 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.624770 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:05.624832 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:46:05.624899 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":3,"thread_start_us":145,"threads_started":1}
I20260806 03:46:05.624982 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.621354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:34681 (local address 127.30.24.1:36363)
0806 03:46:05.621403 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.621405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.621562 (+ 157us) server_negotiation.cc:407] Connection header received
0806 03:46:05.621622 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.621623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.621639 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.621704 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:05.621845 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.622609 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.623913 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.624580 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.624669 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.624682 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.624862 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.624904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:46:05.625449 17021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7919d3e4b05980369a44adf1dd8" candidate_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "befd2ed674dd4459997a525e4d574032" is_pre_election: true
I20260806 03:46:05.625449 17156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7919d3e4b05980369a44adf1dd8" candidate_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee194878204548d49a9c5dd84041e585" is_pre_election: true
I20260806 03:46:05.625610 17156 raft_consensus.cc:2454] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9e2ddfb87f34a7da6216e68a02e14dc in term 0.
I20260806 03:46:05.625610 17021 raft_consensus.cc:2454] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9e2ddfb87f34a7da6216e68a02e14dc in term 0.
I20260806 03:46:05.625891 17227 leader_election.cc:302] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e2ddfb87f34a7da6216e68a02e14dc, ee194878204548d49a9c5dd84041e585; no voters:
I20260806 03:46:05.626012 17361 raft_consensus.cc:2781] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:46:05.626070 17361 raft_consensus.cc:482] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:46:05.626101 17361 raft_consensus.cc:3037] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:05.626806 17361 raft_consensus.cc:504] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.626950 17361 leader_election.cc:288] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [CANDIDATE]: Term 1 election: Requested vote from peers ee194878204548d49a9c5dd84041e585 (127.30.24.2:38319), befd2ed674dd4459997a525e4d574032 (127.30.24.1:36363)
I20260806 03:46:05.627156 17156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7919d3e4b05980369a44adf1dd8" candidate_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee194878204548d49a9c5dd84041e585"
I20260806 03:46:05.627224 17021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7919d3e4b05980369a44adf1dd8" candidate_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "befd2ed674dd4459997a525e4d574032"
I20260806 03:46:05.627259 17156 raft_consensus.cc:3037] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:05.627341 17021 raft_consensus.cc:3037] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:05.628080 17156 raft_consensus.cc:2454] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9e2ddfb87f34a7da6216e68a02e14dc in term 1.
I20260806 03:46:05.628269 17021 raft_consensus.cc:2454] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9e2ddfb87f34a7da6216e68a02e14dc in term 1.
I20260806 03:46:05.628285 17227 leader_election.cc:302] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e2ddfb87f34a7da6216e68a02e14dc, ee194878204548d49a9c5dd84041e585; no voters:
I20260806 03:46:05.628386 17361 raft_consensus.cc:2781] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:05.628556 17361 raft_consensus.cc:686] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 1 LEADER]: Becoming Leader. State: Replica: b9e2ddfb87f34a7da6216e68a02e14dc, State: Running, Role: LEADER
I20260806 03:46:05.628790 17361 consensus_queue.cc:237] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } }
I20260806 03:46:05.629709 16872 catalog_manager.cc:5495] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc reported cstate change: term changed from 0 to 1, leader changed from <none> to b9e2ddfb87f34a7da6216e68a02e14dc (127.30.24.3). New cstate: current_term: 1 leader_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9e2ddfb87f34a7da6216e68a02e14dc" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 } health_report { overall_health: UNKNOWN } } }
W20260806 03:46:05.651960 17339 tablet.cc:2369] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:05.662799 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.660588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.30.24.3:46825)
0806 03:46:05.660650 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.660652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.660663 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:05.660735 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.660736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.660755 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.660811 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:05.660942 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.661418 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.662137 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.662319 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.662368 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.662502 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:05.662588 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:05.662659 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.662714 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.662722 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 03:46:05.664906 17156 raft_consensus.cc:1261] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 1 FOLLOWER]: Refusing update from remote peer b9e2ddfb87f34a7da6216e68a02e14dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:05.664906 17021 raft_consensus.cc:1261] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 1 FOLLOWER]: Refusing update from remote peer b9e2ddfb87f34a7da6216e68a02e14dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:05.665356 17365 consensus_queue.cc:1035] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "befd2ed674dd4459997a525e4d574032" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:05.665499 17361 consensus_queue.cc:1035] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee194878204548d49a9c5dd84041e585" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 38319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:05.668022 17368 mvcc.cc:205] Tried to move back new op lower bound from 7315406707360047104 to 7315406707216224256. Current Snapshot: MvccSnapshot[applied={T|T < 7315406707360047104}]
I20260806 03:46:05.668561 17369 mvcc.cc:205] Tried to move back new op lower bound from 7315406707360047104 to 7315406707216224256. Current Snapshot: MvccSnapshot[applied={T|T < 7315406707360047104}]
I20260806 03:46:05.670820 16872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:05.670907 16872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:05.671646 16872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2212335658c4bcbbf540f807c0f38aa: Sending DeleteTablet for 3 replicas of tablet 283fc7919d3e4b05980369a44adf1dd8
I20260806 03:46:05.672001 17136 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7919d3e4b05980369a44adf1dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:05 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:36114
I20260806 03:46:05.672044 17272 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7919d3e4b05980369a44adf1dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:05 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:39214
I20260806 03:46:05.672266 17379 tablet_replica.cc:330] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: stopping tablet replica
I20260806 03:46:05.672250 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7919d3e4b05980369a44adf1dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:05 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:54868
I20260806 03:46:05.672366 17379 raft_consensus.cc:2229] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:46:05.672487 17379 raft_consensus.cc:2258] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:05.672585 17380 tablet_replica.cc:330] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032: stopping tablet replica
I20260806 03:46:05.672731 17380 raft_consensus.cc:2229] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:05.672895 17380 raft_consensus.cc:2258] T 283fc7919d3e4b05980369a44adf1dd8 P befd2ed674dd4459997a525e4d574032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:05.672941 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16936
I20260806 03:46:05.672940 17379 ts_tablet_manager.cc:1905] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:05.675136 17379 ts_tablet_manager.cc:1918] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:05.675220 17379 log.cc:1196] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/283fc7919d3e4b05980369a44adf1dd8
I20260806 03:46:05.675508 17379 ts_tablet_manager.cc:1939] T 283fc7919d3e4b05980369a44adf1dd8 P b9e2ddfb87f34a7da6216e68a02e14dc: Deleting consensus metadata
I20260806 03:46:05.675954 16858 catalog_manager.cc:4841] TS b9e2ddfb87f34a7da6216e68a02e14dc (127.30.24.3:46825): tablet 283fc7919d3e4b05980369a44adf1dd8 (table test-table [id=40ec796e10e946feb728fb1b1e61a787]) successfully deleted
I20260806 03:46:05.680019 17378 tablet_replica.cc:330] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: stopping tablet replica
I20260806 03:46:05.680151 17378 raft_consensus.cc:2229] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:05.680284 17378 raft_consensus.cc:2258] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:05.680804 17378 ts_tablet_manager.cc:1905] T 283fc7919d3e4b05980369a44adf1dd8 P ee194878204548d49a9c5dd84041e585: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 03:46:05.681123 16859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:36363 (error 108)
I20260806 03:46:05.681639 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.681502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:36363 (local address 127.0.0.1:54876)
0806 03:46:05.681557 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.681605 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:36363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 03:46:05.681753 16859 catalog_manager.cc:4584] TS befd2ed674dd4459997a525e4d574032 (127.30.24.1:36363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 283fc7919d3e4b05980369a44adf1dd8: Network error: Client connection negotiation failed: client connection to 127.30.24.1:36363: connect: Connection refused (error 111)
I20260806 03:46:05.682058 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17072
I20260806 03:46:05.688141 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.688006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:38319 (local address 127.0.0.1:36120)
0806 03:46:05.688070 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:05.688101 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.2:38319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 03:46:05.688221 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17208
I20260806 03:46:05.694794 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 16841
2026-08-06T03:46:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T03:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:05Z Disabled control of system clock
I20260806 03:46:05.724202 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:44947
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:05.806787 17385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:05.807008 17385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:05.807027 17385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:05.808257 17385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:05.808300 17385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:05.808313 17385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:05.808326 17385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:05.808372 17385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:05.809739 17385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:44947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:05.810004 17385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:05.810313 17385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:05.815155 17391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.815172 17390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.815196 17393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:05.815582 17385 server_base.cc:1034] running on GCE node
I20260806 03:46:05.815845 17385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:05.816138 17385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:05.817294 17385 hybrid_clock.cc:648] HybridClock initialized: now 1785987965817277 us; error 39 us; skew 500 ppm
I20260806 03:46:05.818564 17385 webserver.cc:460] Webserver started at http://127.30.24.62:46707/ using document root <none> and password file <none>
I20260806 03:46:05.818832 17385 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:05.818881 17385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:05.819036 17385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:05.820175 17385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "749667b1ef40427497c3773cac2f45a4"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "d82d6c5b44612ecc06b6461f678eb27a"
server_key_iv: "03cd479f0fdad5c516376087a3a76939"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.820582 17385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "749667b1ef40427497c3773cac2f45a4"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "d82d6c5b44612ecc06b6461f678eb27a"
server_key_iv: "03cd479f0fdad5c516376087a3a76939"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.821992 17385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:05.822726 17399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.822911 17385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:05.823002 17385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "749667b1ef40427497c3773cac2f45a4"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "d82d6c5b44612ecc06b6461f678eb27a"
server_key_iv: "03cd479f0fdad5c516376087a3a76939"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.823071 17385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:05.846669 17385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:05.847059 17385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:05.847257 17385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:05.851632 17385 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:44947
I20260806 03:46:05.851670 17454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:44947 every 8 connection(s)
I20260806 03:46:05.852069 17385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:05.853157 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:05.855449 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4: Bootstrap starting.
I20260806 03:46:05.855996 17455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:05.856242 17455 log.cc:824] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:05.856915 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4: No bootstrap required, opened a new log
I20260806 03:46:05.858439 17455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749667b1ef40427497c3773cac2f45a4" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 44947 } }
I20260806 03:46:05.858577 17455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:05.858599 17455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749667b1ef40427497c3773cac2f45a4, State: Initialized, Role: FOLLOWER
I20260806 03:46:05.858760 17455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749667b1ef40427497c3773cac2f45a4" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 44947 } }
I20260806 03:46:05.858857 17455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:05.858865 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17385
I20260806 03:46:05.858892 17455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:05.858946 17455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:05.858985 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:05.859210 30816 external_mini_cluster.cc:1428] Setting key f20746716e4b04e62c9c6c354da49850
I20260806 03:46:05.859623 17455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749667b1ef40427497c3773cac2f45a4" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 44947 } }
I20260806 03:46:05.859761 17455 leader_election.cc:302] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 749667b1ef40427497c3773cac2f45a4; no voters:
I20260806 03:46:05.859936 17455 leader_election.cc:288] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:05.859959 17460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:05.860100 17460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [term 1 LEADER]: Becoming Leader. State: Replica: 749667b1ef40427497c3773cac2f45a4, State: Running, Role: LEADER
I20260806 03:46:05.860210 17455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:05.860195 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749667b1ef40427497c3773cac2f45a4" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 44947 } }
I20260806 03:46:05.860685 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "749667b1ef40427497c3773cac2f45a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749667b1ef40427497c3773cac2f45a4" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 44947 } } }
I20260806 03:46:05.860787 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:05.861204 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 749667b1ef40427497c3773cac2f45a4. Latest consensus state: current_term: 1 leader_uuid: "749667b1ef40427497c3773cac2f45a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749667b1ef40427497c3773cac2f45a4" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 44947 } } }
I20260806 03:46:05.861321 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:05.861199 17469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:05.861938 17469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:05.864056 17469 catalog_manager.cc:1347] Generated new cluster ID: 88daeb91be404317b4c0b92c4c5e5409
I20260806 03:46:05.864112 17469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:05.865373 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:05.859743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.30.24.62:44947)
0806 03:46:05.859966 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:05.859975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:05.860009 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:46:05.860136 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:05.860144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:05.860217 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:05.860699 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:05.860818 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.862162 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.863167 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:05.863739 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:05.863868 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:05.864622 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:05.864626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:05.864626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:05.864810 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:05.864842 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:05.864853 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:05.864924 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:05.865039 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 03:46:05.865214 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260806 03:46:05.881505 17469 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:05.882035 17469 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:05.890916 17469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4: Generated new TSK 0
I20260806 03:46:05.891122 17469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:05.894528 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:44947
--builtin_ntp_servers=127.30.24.20:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:05.978163 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:05.978367 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:05.978389 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:05.979595 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:05.979635 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:05.979665 17479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:46:05.981172 17479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:05.981434 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:05.981760 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:05.987666 17485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.987702 17484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:05.987800 17487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:05.987921 17479 server_base.cc:1034] running on GCE node
I20260806 03:46:05.988211 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:05.988494 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:05.989660 17479 hybrid_clock.cc:648] HybridClock initialized: now 1785987965989647 us; error 28 us; skew 500 ppm
I20260806 03:46:05.990914 17479 webserver.cc:460] Webserver started at http://127.30.24.1:45793/ using document root <none> and password file <none>
I20260806 03:46:05.991194 17479 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:05.991241 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:05.991417 17479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:05.992530 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "e0fa8ccfae7b407d8c7b814705478215"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "8cd6e1874f4dcf3e63f09a96d2a88fb1"
server_key_iv: "c9b212eb21d8810b3119597cb4047307"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.992897 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "e0fa8ccfae7b407d8c7b814705478215"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "8cd6e1874f4dcf3e63f09a96d2a88fb1"
server_key_iv: "c9b212eb21d8810b3119597cb4047307"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.994235 17479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:05.994961 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:05.995136 17479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:05.995247 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "e0fa8ccfae7b407d8c7b814705478215"
format_stamp: "Formatted at 2026-08-06 03:46:05 on dist-test-slave-10pc"
server_key: "8cd6e1874f4dcf3e63f09a96d2a88fb1"
server_key_iv: "c9b212eb21d8810b3119597cb4047307"
server_key_version: "encryptionkey@0"
I20260806 03:46:05.995332 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:06.009480 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:06.009836 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:06.009999 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:06.010249 17479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:06.010586 17479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:06.010644 17479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.010681 17479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:06.010721 17479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.016568 17479 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:38717
I20260806 03:46:06.016651 17607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:38717 every 8 connection(s)
I20260806 03:46:06.017005 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:46:06.020148 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17479
I20260806 03:46:06.020241 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:46:06.020439 30816 external_mini_cluster.cc:1428] Setting key a6fccbad6567e51449dab0bcf882a59b
I20260806 03:46:06.021500 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:44947
--builtin_ntp_servers=127.30.24.20:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:46:06.022864 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.017812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:54843 (local address 127.30.24.62:44947)
0806 03:46:06.017890 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.017893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.018144 (+ 251us) server_negotiation.cc:407] Connection header received
0806 03:46:06.018633 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.018634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.018646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.018709 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:06.019376 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.019881 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.021205 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.021514 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.021593 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.022526 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:06.022528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:06.022528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:06.022534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:06.022551 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:06.022557 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:06.022612 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.022704 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.022738 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:46:06.022950 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.017810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:44947 (local address 127.30.24.1:54843)
0806 03:46:06.018008 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.018050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.018548 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.018763 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.018766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.018774 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:06.019299 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.019303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.019964 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.019965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.021094 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.021098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.021610 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.021611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.021780 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.022382 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:06.022383 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:06.022411 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:06.022432 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:06.022619 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:06.022624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:06.022626 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.022681 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.022762 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260806 03:46:06.023553 17608 heartbeater.cc:347] Connected to a master server at 127.30.24.62:44947
I20260806 03:46:06.023644 17608 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:06.023849 17608 heartbeater.cc:511] Master 127.30.24.62:44947 requested a full tablet report, sending...
I20260806 03:46:06.024492 17417 ts_manager.cc:194] Registered new tserver with Master: e0fa8ccfae7b407d8c7b814705478215 (127.30.24.1:38717)
I20260806 03:46:06.025100 17417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.1:54843
W20260806 03:46:06.106074 17611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:06.106278 17611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:06.106299 17611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:06.107349 17611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:06.107380 17611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:06.107404 17611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:46:06.108810 17611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:06.108996 17611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:06.109247 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:06.114200 17616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:06.114179 17619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:06.114252 17611 server_base.cc:1034] running on GCE node
W20260806 03:46:06.114177 17617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:06.114617 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:06.114876 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:06.116039 17611 hybrid_clock.cc:648] HybridClock initialized: now 1785987966116013 us; error 36 us; skew 500 ppm
I20260806 03:46:06.117286 17611 webserver.cc:460] Webserver started at http://127.30.24.2:37417/ using document root <none> and password file <none>
I20260806 03:46:06.117528 17611 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:06.117596 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:06.117761 17611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:06.118870 17611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3"
format_stamp: "Formatted at 2026-08-06 03:46:06 on dist-test-slave-10pc"
server_key: "777635220c64ffbfedfefff7f9610872"
server_key_iv: "6a739d765469d88bdbd28582331c1c03"
server_key_version: "encryptionkey@0"
I20260806 03:46:06.119247 17611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3"
format_stamp: "Formatted at 2026-08-06 03:46:06 on dist-test-slave-10pc"
server_key: "777635220c64ffbfedfefff7f9610872"
server_key_iv: "6a739d765469d88bdbd28582331c1c03"
server_key_version: "encryptionkey@0"
I20260806 03:46:06.120584 17611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:06.121327 17626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.121490 17611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:06.121590 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3"
format_stamp: "Formatted at 2026-08-06 03:46:06 on dist-test-slave-10pc"
server_key: "777635220c64ffbfedfefff7f9610872"
server_key_iv: "6a739d765469d88bdbd28582331c1c03"
server_key_version: "encryptionkey@0"
I20260806 03:46:06.121683 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:06.135601 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:06.135927 17611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:06.136094 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:06.136345 17611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:06.136696 17611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:06.136751 17611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.136786 17611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:06.136833 17611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.142846 17611 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:40201
I20260806 03:46:06.142903 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:40201 every 8 connection(s)
I20260806 03:46:06.143232 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:46:06.148299 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.143979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:49593 (local address 127.30.24.62:44947)
0806 03:46:06.144049 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.144051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.144319 (+ 268us) server_negotiation.cc:407] Connection header received
0806 03:46:06.144836 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.144837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.144846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.144905 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:06.145622 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.146127 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.147077 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.147281 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.147337 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.148065 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:06.148066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:06.148066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:06.148070 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:06.148085 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:06.148089 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:06.148135 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.148212 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.148239 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 03:46:06.148494 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.143975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:44947 (local address 127.30.24.2:49593)
0806 03:46:06.144197 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.144241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.144749 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.144959 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.144963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.144971 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:06.145517 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.145522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.146216 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.146216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.146981 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.146982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.147357 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.147358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.147466 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.147935 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:06.147935 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:06.147964 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:06.147986 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:06.148138 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:06.148140 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:06.148142 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.148199 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.148276 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260806 03:46:06.148687 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17611
I20260806 03:46:06.148805 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:46:06.149004 30816 external_mini_cluster.cc:1428] Setting key 5d5c1f08264ed595c7d4d5ddd34b2258
I20260806 03:46:06.149021 17744 heartbeater.cc:347] Connected to a master server at 127.30.24.62:44947
I20260806 03:46:06.149142 17744 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:06.149334 17744 heartbeater.cc:511] Master 127.30.24.62:44947 requested a full tablet report, sending...
I20260806 03:46:06.149753 17417 ts_manager.cc:194] Registered new tserver with Master: ca78ab99f1e34cff8b80f91dfca1a0a3 (127.30.24.2:40201)
I20260806 03:46:06.150074 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:44947
--builtin_ntp_servers=127.30.24.20:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:46:06.150132 17417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.2:49593
W20260806 03:46:06.235635 17747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:06.235836 17747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:06.235857 17747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:06.237524 17747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:06.237581 17747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:06.237627 17747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:46:06.239809 17747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:06.240084 17747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:06.240509 17747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:06.247417 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:06.247458 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:06.247453 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:06.247756 17747 server_base.cc:1034] running on GCE node
I20260806 03:46:06.247964 17747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:06.248191 17747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:06.249332 17747 hybrid_clock.cc:648] HybridClock initialized: now 1785987966249320 us; error 27 us; skew 500 ppm
I20260806 03:46:06.250643 17747 webserver.cc:460] Webserver started at http://127.30.24.3:42885/ using document root <none> and password file <none>
I20260806 03:46:06.250903 17747 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:06.250949 17747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:06.251104 17747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:06.252240 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "31477e8d4b9c4667abdce023a4de13b4"
format_stamp: "Formatted at 2026-08-06 03:46:06 on dist-test-slave-10pc"
server_key: "cc2e1812af0a91170d23e4c7afa2dc91"
server_key_iv: "c8b33902ffb2100f1ec13447a40e4f4a"
server_key_version: "encryptionkey@0"
I20260806 03:46:06.252647 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "31477e8d4b9c4667abdce023a4de13b4"
format_stamp: "Formatted at 2026-08-06 03:46:06 on dist-test-slave-10pc"
server_key: "cc2e1812af0a91170d23e4c7afa2dc91"
server_key_iv: "c8b33902ffb2100f1ec13447a40e4f4a"
server_key_version: "encryptionkey@0"
I20260806 03:46:06.253986 17747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:06.254841 17761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.255034 17747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:06.255125 17747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "31477e8d4b9c4667abdce023a4de13b4"
format_stamp: "Formatted at 2026-08-06 03:46:06 on dist-test-slave-10pc"
server_key: "cc2e1812af0a91170d23e4c7afa2dc91"
server_key_iv: "c8b33902ffb2100f1ec13447a40e4f4a"
server_key_version: "encryptionkey@0"
I20260806 03:46:06.255198 17747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:06.290969 17747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:06.291314 17747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:06.291469 17747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:06.291747 17747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:06.292102 17747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:06.292137 17747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.292162 17747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:06.292208 17747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:06.297889 17747 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:41439
I20260806 03:46:06.297968 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:41439 every 8 connection(s)
I20260806 03:46:06.298342 17747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:46:06.303654 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.299062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:49741 (local address 127.30.24.62:44947)
0806 03:46:06.299107 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.299109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.299400 (+ 291us) server_negotiation.cc:407] Connection header received
0806 03:46:06.299951 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.299952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.299965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.300021 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:06.300808 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.301312 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.302305 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.302516 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.302589 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.303381 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:06.303382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:06.303383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:06.303388 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:06.303401 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:06.303406 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:06.303454 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.303542 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.303574 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:46:06.303893 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.299046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:44947 (local address 127.30.24.3:49741)
0806 03:46:06.299266 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.299311 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.299838 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.300093 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.300096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.300106 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:06.300708 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.300714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.301406 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.301407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.302197 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.302199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.302599 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.302600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.302740 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.303242 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:06.303242 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:06.303270 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:06.303293 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:06.303460 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:06.303465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:06.303467 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.303544 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.303644 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260806 03:46:06.304400 17877 heartbeater.cc:347] Connected to a master server at 127.30.24.62:44947
I20260806 03:46:06.304513 17877 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:06.304678 17877 heartbeater.cc:511] Master 127.30.24.62:44947 requested a full tablet report, sending...
I20260806 03:46:06.305063 17417 ts_manager.cc:194] Registered new tserver with Master: 31477e8d4b9c4667abdce023a4de13b4 (127.30.24.3:41439)
I20260806 03:46:06.305423 17417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.3:49741
I20260806 03:46:06.306335 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17747
I20260806 03:46:06.306423 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:46:06.306641 30816 external_mini_cluster.cc:1428] Setting key e60432388520bb3d2709ceed8588f6bb
I20260806 03:46:06.307514 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:46:06.311708 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.308929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.30.24.62:44947)
0806 03:46:06.308978 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.308980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.309038 (+ 58us) server_negotiation.cc:407] Connection header received
0806 03:46:06.309099 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.309100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.309109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.309146 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:06.309316 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.309764 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.310467 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.310745 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.310800 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.311432 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:06.311434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:06.311434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:06.311438 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:06.311451 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:06.311455 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:06.311527 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.311590 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.311600 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:46:06.313238 17417 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:46:06.314738 17417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:46:06.321224 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.316983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.30.24.2:40201)
0806 03:46:06.317554 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.317558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.317570 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:46:06.317607 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.317609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.317623 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.317723 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:06.317844 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.318603 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.319958 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.320820 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.320914 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.321007 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.321098 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.321134 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":95,"threads_started":1}
I20260806 03:46:06.321334 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.316893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:38717 (local address 127.0.0.1:34522)
0806 03:46:06.317183 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.317194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.317251 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.317642 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.317646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.317649 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:06.317720 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.317722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.319086 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.319086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.320067 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.320069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.321119 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.321120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.321173 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.321176 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.321222 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.321256 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260806 03:46:06.321532 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.317147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.30.24.3:41439)
0806 03:46:06.317351 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.317357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.317369 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:46:06.317472 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.317475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.317490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.317918 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:06.318044 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.319109 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.320259 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.320940 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.321300 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.321392 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.321420 (+ 28us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.321459 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260806 03:46:06.321568 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.317153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.30.24.1:38717)
0806 03:46:06.317468 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.317472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.317482 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:46:06.317523 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.317525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.317541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.317619 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:06.317812 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.318987 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.320151 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.320912 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.320992 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.321084 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.321445 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.321481 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":89,"threads_started":1}
I20260806 03:46:06.321261 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.316887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:40201 (local address 127.0.0.1:44878)
0806 03:46:06.317083 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.317123 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.317203 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.317727 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.317728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.317730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:06.317777 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.317779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.318687 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.318688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.319868 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.319870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.320916 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.320917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.320994 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.321007 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.321084 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.321137 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260806 03:46:06.321784 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.316918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:41439 (local address 127.0.0.1:38884)
0806 03:46:06.317317 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.317327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.317365 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.317931 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.317931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.317933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:06.317966 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.317967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.319194 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.319195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.320170 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.320173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.321035 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.321036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.321097 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.321103 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.321135 (+ 32us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.321187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":84,"threads_started":1}
I20260806 03:46:06.322324 17542 tablet_service.cc:1467] Processing CreateTablet for tablet 36411639374c4d668b9fbd5193eef298 (DEFAULT_TABLE table=test-table [id=73e546dca72840aca48fa304b23aef7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:06.322410 17676 tablet_service.cc:1467] Processing CreateTablet for tablet 36411639374c4d668b9fbd5193eef298 (DEFAULT_TABLE table=test-table [id=73e546dca72840aca48fa304b23aef7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:06.322731 17676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36411639374c4d668b9fbd5193eef298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:06.322731 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36411639374c4d668b9fbd5193eef298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:06.322789 17809 tablet_service.cc:1467] Processing CreateTablet for tablet 36411639374c4d668b9fbd5193eef298 (DEFAULT_TABLE table=test-table [id=73e546dca72840aca48fa304b23aef7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:06.323133 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36411639374c4d668b9fbd5193eef298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:06.326217 17891 tablet_bootstrap.cc:492] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Bootstrap starting.
I20260806 03:46:06.326563 17893 tablet_bootstrap.cc:492] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: Bootstrap starting.
I20260806 03:46:06.327039 17891 tablet_bootstrap.cc:654] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:06.327167 17893 tablet_bootstrap.cc:654] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:06.327365 17891 log.cc:824] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:06.327471 17893 log.cc:824] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:06.327838 17892 tablet_bootstrap.cc:492] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Bootstrap starting.
I20260806 03:46:06.328125 17891 tablet_bootstrap.cc:492] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: No bootstrap required, opened a new log
I20260806 03:46:06.328190 17891 ts_tablet_manager.cc:1397] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:46:06.328495 17892 tablet_bootstrap.cc:654] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:06.328778 17892 log.cc:824] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:06.328850 17893 tablet_bootstrap.cc:492] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: No bootstrap required, opened a new log
I20260806 03:46:06.328933 17893 ts_tablet_manager.cc:1397] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:46:06.329454 17892 tablet_bootstrap.cc:492] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: No bootstrap required, opened a new log
I20260806 03:46:06.329537 17892 ts_tablet_manager.cc:1397] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:46:06.330960 17891 raft_consensus.cc:348] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.331146 17891 raft_consensus.cc:374] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:06.331194 17891 raft_consensus.cc:729] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31477e8d4b9c4667abdce023a4de13b4, State: Initialized, Role: FOLLOWER
I20260806 03:46:06.331358 17892 raft_consensus.cc:348] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.331503 17892 raft_consensus.cc:374] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:06.331545 17892 raft_consensus.cc:729] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca78ab99f1e34cff8b80f91dfca1a0a3, State: Initialized, Role: FOLLOWER
I20260806 03:46:06.331526 17893 raft_consensus.cc:348] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.331703 17893 raft_consensus.cc:374] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:06.331671 17892 consensus_queue.cc:260] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.331754 17893 raft_consensus.cc:729] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa8ccfae7b407d8c7b814705478215, State: Initialized, Role: FOLLOWER
I20260806 03:46:06.331735 17891 consensus_queue.cc:260] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.331890 17893 consensus_queue.cc:260] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.331946 17892 ts_tablet_manager.cc:1428] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 03:46:06.331980 17744 heartbeater.cc:503] Master 127.30.24.62:44947 was elected leader, sending a full tablet report...
I20260806 03:46:06.332003 17891 ts_tablet_manager.cc:1428] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260806 03:46:06.332135 17893 ts_tablet_manager.cc:1428] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 03:46:06.332191 17608 heartbeater.cc:503] Master 127.30.24.62:44947 was elected leader, sending a full tablet report...
I20260806 03:46:06.332265 17877 heartbeater.cc:503] Master 127.30.24.62:44947 was elected leader, sending a full tablet report...
W20260806 03:46:06.393772 17745 tablet.cc:2369] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:46:06.517760 17609 tablet.cc:2369] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:46:06.548909 17878 tablet.cc:2369] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:06.599717 17899 raft_consensus.cc:482] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:46:06.599897 17899 raft_consensus.cc:504] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.600330 17899 leader_election.cc:288] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca78ab99f1e34cff8b80f91dfca1a0a3 (127.30.24.2:40201), 31477e8d4b9c4667abdce023a4de13b4 (127.30.24.3:41439)
I20260806 03:46:06.603703 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.600625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:60055 (local address 127.30.24.3:41439)
0806 03:46:06.600692 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.600695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.600800 (+ 105us) server_negotiation.cc:407] Connection header received
0806 03:46:06.600883 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.600884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.600891 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.600958 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:06.601154 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.601595 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.602756 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.603307 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.603394 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.603405 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.603602 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.603643 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:46:06.603698 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.600671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:43857 (local address 127.30.24.2:40201)
0806 03:46:06.600730 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.600731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.600745 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:46:06.600865 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.600867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.600881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.600958 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:06.601126 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.601602 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.602795 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.603330 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.603395 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.603405 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.603585 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.603628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:06.603724 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.600583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:41439 (local address 127.30.24.1:60055)
0806 03:46:06.600715 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.600727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.600800 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.600965 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.600966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.600968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:06.601062 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.601064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.601683 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.601684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.602682 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.602684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.603413 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.603414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.603495 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.603502 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.603564 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.603613 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":71,"thread_start_us":59,"threads_started":1}
I20260806 03:46:06.603729 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.600553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:40201 (local address 127.30.24.1:43857)
0806 03:46:06.600687 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.600700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:06.600763 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:06.600969 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:06.600970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:06.600972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:06.601052 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.601054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.601708 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.601709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.602681 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:06.602684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.603415 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.603416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:06.603495 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.603503 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 03:46:06.603564 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 03:46:06.603613 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":7,"thread_start_us":66,"threads_started":1}
I20260806 03:46:06.604099 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36411639374c4d668b9fbd5193eef298" candidate_uuid: "e0fa8ccfae7b407d8c7b814705478215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" is_pre_election: true
I20260806 03:46:06.604115 17831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36411639374c4d668b9fbd5193eef298" candidate_uuid: "e0fa8ccfae7b407d8c7b814705478215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31477e8d4b9c4667abdce023a4de13b4" is_pre_election: true
I20260806 03:46:06.604269 17697 raft_consensus.cc:2454] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0fa8ccfae7b407d8c7b814705478215 in term 0.
I20260806 03:46:06.604269 17831 raft_consensus.cc:2454] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0fa8ccfae7b407d8c7b814705478215 in term 0.
I20260806 03:46:06.604522 17494 leader_election.cc:302] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca78ab99f1e34cff8b80f91dfca1a0a3, e0fa8ccfae7b407d8c7b814705478215; no voters:
I20260806 03:46:06.604712 17899 raft_consensus.cc:2781] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:46:06.604753 17899 raft_consensus.cc:482] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:46:06.604768 17899 raft_consensus.cc:3037] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:06.605460 17899 raft_consensus.cc:504] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.605616 17899 leader_election.cc:288] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [CANDIDATE]: Term 1 election: Requested vote from peers ca78ab99f1e34cff8b80f91dfca1a0a3 (127.30.24.2:40201), 31477e8d4b9c4667abdce023a4de13b4 (127.30.24.3:41439)
I20260806 03:46:06.605769 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36411639374c4d668b9fbd5193eef298" candidate_uuid: "e0fa8ccfae7b407d8c7b814705478215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3"
I20260806 03:46:06.605826 17831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36411639374c4d668b9fbd5193eef298" candidate_uuid: "e0fa8ccfae7b407d8c7b814705478215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31477e8d4b9c4667abdce023a4de13b4"
I20260806 03:46:06.605866 17697 raft_consensus.cc:3037] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:06.605922 17831 raft_consensus.cc:3037] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:06.606653 17697 raft_consensus.cc:2454] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0fa8ccfae7b407d8c7b814705478215 in term 1.
I20260806 03:46:06.606655 17831 raft_consensus.cc:2454] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0fa8ccfae7b407d8c7b814705478215 in term 1.
I20260806 03:46:06.606853 17497 leader_election.cc:302] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31477e8d4b9c4667abdce023a4de13b4, e0fa8ccfae7b407d8c7b814705478215; no voters:
I20260806 03:46:06.606945 17899 raft_consensus.cc:2781] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:06.607255 17899 raft_consensus.cc:686] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [term 1 LEADER]: Becoming Leader. State: Replica: e0fa8ccfae7b407d8c7b814705478215, State: Running, Role: LEADER
I20260806 03:46:06.607366 17899 consensus_queue.cc:237] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } }
I20260806 03:46:06.608372 17417 catalog_manager.cc:5495] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0fa8ccfae7b407d8c7b814705478215 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "e0fa8ccfae7b407d8c7b814705478215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa8ccfae7b407d8c7b814705478215" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 38717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 } health_report { overall_health: UNKNOWN } } }
I20260806 03:46:06.677606 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.675217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.30.24.1:38717)
0806 03:46:06.675283 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:06.675285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:06.675293 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:06.675331 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:06.675332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:06.675355 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:06.675418 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:06.675565 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.676035 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.676772 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:06.676963 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:06.677026 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:06.677154 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:06.677241 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:06.677318 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:06.677487 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 03:46:06.677501 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 03:46:06.679548 17831 raft_consensus.cc:1261] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 1 FOLLOWER]: Refusing update from remote peer e0fa8ccfae7b407d8c7b814705478215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:06.679549 17697 raft_consensus.cc:1261] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 1 FOLLOWER]: Refusing update from remote peer e0fa8ccfae7b407d8c7b814705478215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:06.679961 17899 consensus_queue.cc:1035] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31477e8d4b9c4667abdce023a4de13b4" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:06.680119 17902 consensus_queue.cc:1035] T 36411639374c4d668b9fbd5193eef298 P e0fa8ccfae7b407d8c7b814705478215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca78ab99f1e34cff8b80f91dfca1a0a3" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:06.683449 17907 mvcc.cc:205] Tried to move back new op lower bound from 7315406711516577792 to 7315406711224471552. Current Snapshot: MvccSnapshot[applied={T|T < 7315406711516577792}]
I20260806 03:46:06.684825 17414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:06.684922 17414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:06.685752 17414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 749667b1ef40427497c3773cac2f45a4: Sending DeleteTablet for 3 replicas of tablet 36411639374c4d668b9fbd5193eef298
I20260806 03:46:06.686086 17809 tablet_service.cc:1514] Processing DeleteTablet for tablet 36411639374c4d668b9fbd5193eef298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:06 UTC) from {username='slave'} at 127.0.0.1:38884
I20260806 03:46:06.686089 17676 tablet_service.cc:1514] Processing DeleteTablet for tablet 36411639374c4d668b9fbd5193eef298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:06 UTC) from {username='slave'} at 127.0.0.1:44878
I20260806 03:46:06.686381 17915 tablet_replica.cc:330] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: stopping tablet replica
I20260806 03:46:06.686379 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet 36411639374c4d668b9fbd5193eef298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:06 UTC) from {username='slave'} at 127.0.0.1:34522
I20260806 03:46:06.686481 17915 raft_consensus.cc:2229] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:06.686604 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17479
I20260806 03:46:06.686642 17915 raft_consensus.cc:2258] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:06.688709 17916 tablet_replica.cc:330] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: stopping tablet replica
I20260806 03:46:06.688787 17916 raft_consensus.cc:2229] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:06.688858 17916 raft_consensus.cc:2258] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:06.689294 17916 ts_tablet_manager.cc:1905] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:06.689296 17915 ts_tablet_manager.cc:1905] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:06.691121 17916 ts_tablet_manager.cc:1918] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:06.691121 17915 ts_tablet_manager.cc:1918] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:06.691206 17916 log.cc:1196] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/36411639374c4d668b9fbd5193eef298
I20260806 03:46:06.691206 17915 log.cc:1196] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/36411639374c4d668b9fbd5193eef298
I20260806 03:46:06.691462 17916 ts_tablet_manager.cc:1939] T 36411639374c4d668b9fbd5193eef298 P ca78ab99f1e34cff8b80f91dfca1a0a3: Deleting consensus metadata
I20260806 03:46:06.691462 17915 ts_tablet_manager.cc:1939] T 36411639374c4d668b9fbd5193eef298 P 31477e8d4b9c4667abdce023a4de13b4: Deleting consensus metadata
I20260806 03:46:06.691869 17403 catalog_manager.cc:4841] TS 31477e8d4b9c4667abdce023a4de13b4 (127.30.24.3:41439): tablet 36411639374c4d668b9fbd5193eef298 (table test-table [id=73e546dca72840aca48fa304b23aef7d]) successfully deleted
I20260806 03:46:06.691916 17400 catalog_manager.cc:4841] TS ca78ab99f1e34cff8b80f91dfca1a0a3 (127.30.24.2:40201): tablet 36411639374c4d668b9fbd5193eef298 (table test-table [id=73e546dca72840aca48fa304b23aef7d]) successfully deleted
W20260806 03:46:06.693943 17401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:38717 (error 108)
I20260806 03:46:06.694090 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17611
I20260806 03:46:06.694921 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:06.694272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:38717 (local address 127.0.0.1:37996)
0806 03:46:06.694839 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:06.694879 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:38717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547}
W20260806 03:46:06.696058 17401 catalog_manager.cc:4584] TS e0fa8ccfae7b407d8c7b814705478215 (127.30.24.1:38717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36411639374c4d668b9fbd5193eef298: Network error: Client connection negotiation failed: client connection to 127.30.24.1:38717: connect: Connection refused (error 111)
I20260806 03:46:06.699569 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17747
I20260806 03:46:06.704319 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17385
2026-08-06T03:46:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:06 dist-test-slave-10pc krb5kdc[17921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:06 dist-test-slave-10pc krb5kdc[17921](info): set up 2 sockets
Aug 06 03:46:06 dist-test-slave-10pc krb5kdc[17921](info): commencing operation
krb5kdc: starting...
W20260806 03:46:07.777454 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:07 dist-test-slave-10pc krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:07.900863 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:46447
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:46447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:07.982826 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:07.983047 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:07.983088 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:07.984299 17934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:07.984349 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:07.984362 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:07.984375 17934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:07.984431 17934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:07.986102 17934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:46447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:46447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:07.986384 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:07.986630 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:07.991709 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:07.991756 17934 server_base.cc:1034] running on GCE node
W20260806 03:46:07.991720 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:07.991720 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:07.992132 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:07.992427 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:07.993623 17934 hybrid_clock.cc:648] HybridClock initialized: now 1785987967993611 us; error 30 us; skew 500 ppm
Aug 06 03:46:07 dist-test-slave-10pc krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:07.996201 17934 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:07.996672 17934 webserver.cc:460] Webserver started at http://127.30.24.62:38935/ using document root <none> and password file <none>
I20260806 03:46:07.996881 17934 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:07.996924 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:07.997030 17934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:07.998057 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "c99b4c3f039a4829a86e9f9bb287f7f9"
format_stamp: "Formatted at 2026-08-06 03:46:07 on dist-test-slave-10pc"
server_key: "80bd9a82271a3581c8a35b03995cf7b1"
server_key_iv: "fc696a2e3614f5207bc6ecc27cc9a7a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:07.998385 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "c99b4c3f039a4829a86e9f9bb287f7f9"
format_stamp: "Formatted at 2026-08-06 03:46:07 on dist-test-slave-10pc"
server_key: "80bd9a82271a3581c8a35b03995cf7b1"
server_key_iv: "fc696a2e3614f5207bc6ecc27cc9a7a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:07.999665 17934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:08.000365 17950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.000560 17934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:08.000648 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "c99b4c3f039a4829a86e9f9bb287f7f9"
format_stamp: "Formatted at 2026-08-06 03:46:07 on dist-test-slave-10pc"
server_key: "80bd9a82271a3581c8a35b03995cf7b1"
server_key_iv: "fc696a2e3614f5207bc6ecc27cc9a7a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.000707 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:08.008522 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:08.010812 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:08.010924 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:08.014626 17934 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:46447
I20260806 03:46:08.014945 18003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:46447 every 8 connection(s)
I20260806 03:46:08.015003 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:08.015751 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:08.015849 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 17934
I20260806 03:46:08.015972 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:08.016222 30816 external_mini_cluster.cc:1428] Setting key aa97b0a80d301fabe2897129b376dd9b
I20260806 03:46:08.019313 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9: Bootstrap starting.
I20260806 03:46:08.020004 18004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:08.020292 18004 log.cc:824] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:08.021126 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9: No bootstrap required, opened a new log
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:08.023769 18004 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46447 } }
I20260806 03:46:08.023981 18004 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:08.024022 18004 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99b4c3f039a4829a86e9f9bb287f7f9, State: Initialized, Role: FOLLOWER
I20260806 03:46:08.024180 18004 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46447 } }
I20260806 03:46:08.024286 18004 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:08.024324 18004 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:08.024377 18004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:08.025187 18004 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46447 } }
I20260806 03:46:08.025302 18004 leader_election.cc:302] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99b4c3f039a4829a86e9f9bb287f7f9; no voters:
I20260806 03:46:08.025455 18004 leader_election.cc:288] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:08.025537 18009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:08.025703 18009 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [term 1 LEADER]: Becoming Leader. State: Replica: c99b4c3f039a4829a86e9f9bb287f7f9, State: Running, Role: LEADER
I20260806 03:46:08.025729 18004 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:08.025794 18009 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46447 } }
I20260806 03:46:08.026098 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46447 } } }
I20260806 03:46:08.026206 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:08.026445 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99b4c3f039a4829a86e9f9bb287f7f9. Latest consensus state: current_term: 1 leader_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99b4c3f039a4829a86e9f9bb287f7f9" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 46447 } } }
I20260806 03:46:08.026583 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:08.026857 18015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:08.027366 18015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:08.027681 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.016836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.30.24.62:46447)
0806 03:46:08.017027 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.017037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.017067 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:46:08.017540 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.017550 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.017612 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.017764 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:08.018351 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.019719 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.020562 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.021101 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.021188 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.023790 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:08.023794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:08.023795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:08.023807 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:08.025844 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.026312 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.026315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.026316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.026438 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.026672 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.026673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.026673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.027028 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:08.027206 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.027322 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.027475 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":58,"threads_started":1}
I20260806 03:46:08.029095 18015 catalog_manager.cc:1347] Generated new cluster ID: 28b592873aad473886bfc27e358a7652
I20260806 03:46:08.029156 18015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:08.059283 18015 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:08.059863 18015 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:08.063591 18015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9: Generated new TSK 0
I20260806 03:46:08.063818 18015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.24.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
Entry for principal kudu/127.30.24.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab.
I20260806 03:46:08.099036 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46447
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:08.183259 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:08.183468 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:08.183498 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:08.184667 18030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:08.184710 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:08.184737 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:46:08.186298 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.1.keytab
--principal=kudu/127.30.24.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:08.186496 18030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:08.186674 18030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:08.192065 18038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:08.192108 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:08.192060 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:08.192561 18030 server_base.cc:1034] running on GCE node
I20260806 03:46:08.192780 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:08.193063 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:08.194211 18030 hybrid_clock.cc:648] HybridClock initialized: now 1785987968194200 us; error 32 us; skew 500 ppm
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:08.196758 18030 init.cc:377] Logged in from keytab as kudu/127.30.24.1@KRBTEST.COM (short username kudu)
I20260806 03:46:08.197336 18030 webserver.cc:460] Webserver started at http://127.30.24.1:37641/ using document root <none> and password file <none>
I20260806 03:46:08.197590 18030 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:08.197661 18030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:08.197814 18030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:08.198885 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "203dea29ff054dce9c546f4e0b3953a2"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "3212e56daac456a10a491b95d35708a7"
server_key_iv: "257919fab5d1dd97190d6f914b3641b2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.199298 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "203dea29ff054dce9c546f4e0b3953a2"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "3212e56daac456a10a491b95d35708a7"
server_key_iv: "257919fab5d1dd97190d6f914b3641b2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.200877 18030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260806 03:46:08.201663 18045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.201875 18030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:08.201993 18030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "203dea29ff054dce9c546f4e0b3953a2"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "3212e56daac456a10a491b95d35708a7"
server_key_iv: "257919fab5d1dd97190d6f914b3641b2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.202080 18030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:08.220515 18030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:08.222982 18030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:08.223165 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:08.223414 18030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:08.223747 18030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:08.223801 18030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.223843 18030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:08.223872 18030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.230753 18030 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:34105
I20260806 03:46:08.230808 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:34105 every 8 connection(s)
I20260806 03:46:08.231161 18030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:46:08.234468 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18030
I20260806 03:46:08.234592 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:46:08.234787 30816 external_mini_cluster.cc:1428] Setting key 1838cf4780ee7c8b206331bff97d228d
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.1@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:08.242017 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.231885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:58753 (local address 127.30.24.62:46447)
0806 03:46:08.231949 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.231951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.232151 (+ 200us) server_negotiation.cc:407] Connection header received
0806 03:46:08.232658 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.232659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.232669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.232728 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:08.233670 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.234191 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.235678 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.235885 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.235944 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.238727 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:08.238729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:08.238729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:08.238737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:08.240541 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.241010 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.241011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.241012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.241068 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.241335 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.241336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.241336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.241460 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:08.241501 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.241842 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.241926 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:08.242620 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.231835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46447 (local address 127.30.24.1:58753)
0806 03:46:08.232020 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.232065 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.232572 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.232782 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.232785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.233351 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:08.233556 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.233561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.234320 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.234321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.235586 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.235590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.235978 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.235979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.236122 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.236735 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:08.236746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:08.238619 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:08.240633 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:08.240640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:08.240643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:08.240922 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:08.241137 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:08.241137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:08.241137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:08.241264 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:08.241544 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:08.241550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:08.241701 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.242220 (+ 519us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.242397 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260806 03:46:08.243897 18162 heartbeater.cc:347] Connected to a master server at 127.30.24.62:46447
I20260806 03:46:08.244005 18162 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:08.244235 18162 heartbeater.cc:511] Master 127.30.24.62:46447 requested a full tablet report, sending...
I20260806 03:46:08.245028 17968 ts_manager.cc:194] Registered new tserver with Master: 203dea29ff054dce9c546f4e0b3953a2 (127.30.24.1:34105)
I20260806 03:46:08.246382 17968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.1@KRBTEST.COM'} at 127.30.24.1:58753
WARNING: no policy specified for kudu/127.30.24.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
Entry for principal kudu/127.30.24.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab.
I20260806 03:46:08.274593 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46447
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:08.358811 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:08.359043 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:08.359068 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:08.360203 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:08.360239 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:08.360265 18167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:46:08.361692 18167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.2.keytab
--principal=kudu/127.30.24.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:08.361887 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:08.362082 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:08.367157 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:08.367138 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:08.367214 18167 server_base.cc:1034] running on GCE node
W20260806 03:46:08.367138 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:08.367547 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:08.367761 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:08.368930 18167 hybrid_clock.cc:648] HybridClock initialized: now 1785987968368916 us; error 44 us; skew 500 ppm
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:08.371500 18167 init.cc:377] Logged in from keytab as kudu/127.30.24.2@KRBTEST.COM (short username kudu)
I20260806 03:46:08.371953 18167 webserver.cc:460] Webserver started at http://127.30.24.2:37915/ using document root <none> and password file <none>
I20260806 03:46:08.372156 18167 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:08.372205 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:08.372294 18167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:08.373373 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "57688024631d48eba5fd061ec86214be"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "e728271147d5c0be1e8d3c07cfca46e3"
server_key_iv: "b5108e57a7627f046fec38a43d7761e2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.373683 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "57688024631d48eba5fd061ec86214be"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "e728271147d5c0be1e8d3c07cfca46e3"
server_key_iv: "b5108e57a7627f046fec38a43d7761e2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.374920 18167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:08.375630 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.375892 18167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:08.375979 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "57688024631d48eba5fd061ec86214be"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "e728271147d5c0be1e8d3c07cfca46e3"
server_key_iv: "b5108e57a7627f046fec38a43d7761e2"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.376034 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:08.394397 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:08.396258 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:08.396371 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:08.396620 18167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:08.396955 18167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:08.396989 18167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.397012 18167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:08.397058 18167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.402832 18167 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:36889
I20260806 03:46:08.402904 18298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:36889 every 8 connection(s)
I20260806 03:46:08.403303 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.2@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:08.409559 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18167
I20260806 03:46:08.409682 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:46:08.409906 30816 external_mini_cluster.cc:1428] Setting key cd020d3b6dffea9434a7162de5e06cc9
I20260806 03:46:08.413312 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.403967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46447 (local address 127.30.24.2:41067)
0806 03:46:08.404149 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.404187 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.404738 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.404944 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.404947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.405548 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:08.405755 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.405759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.406513 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.406514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.407322 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.407324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.407652 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.407653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.407765 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.408254 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:08.408263 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:08.409863 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:08.411703 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:08.411710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:08.411713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:08.412056 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:08.412331 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:08.412333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:08.412333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:08.412515 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:08.412825 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:08.412831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:08.412949 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.413025 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.413150 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260806 03:46:08.413358 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.404005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:41067 (local address 127.30.24.62:46447)
0806 03:46:08.404083 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.404086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.404324 (+ 238us) server_negotiation.cc:407] Connection header received
0806 03:46:08.404828 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.404829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.404839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.404892 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:08.405835 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.406430 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.407397 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.407580 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.407629 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.409987 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:08.409989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:08.409989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:08.409999 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:08.411608 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.412152 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.412153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.412154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.412215 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.412591 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.412592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.412593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.412754 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:08.412832 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.413133 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.413233 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 03:46:08.413784 18299 heartbeater.cc:347] Connected to a master server at 127.30.24.62:46447
I20260806 03:46:08.413894 18299 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:08.414083 18299 heartbeater.cc:511] Master 127.30.24.62:46447 requested a full tablet report, sending...
I20260806 03:46:08.414517 17968 ts_manager.cc:194] Registered new tserver with Master: 57688024631d48eba5fd061ec86214be (127.30.24.2:36889)
I20260806 03:46:08.414891 17968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.2@KRBTEST.COM'} at 127.30.24.2:41067
WARNING: no policy specified for kudu/127.30.24.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
Entry for principal kudu/127.30.24.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab.
I20260806 03:46:08.437525 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46447
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:08.522222 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:08.522456 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:08.522496 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:08.523620 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:08.523686 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:08.523733 18304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:46:08.525484 18304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.3.keytab
--principal=kudu/127.30.24.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:46447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:08.525758 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:08.526041 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:08.531301 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:08.531301 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:08.531570 18304 server_base.cc:1034] running on GCE node
W20260806 03:46:08.531317 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:08.531898 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:08.532151 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:08.533301 18304 hybrid_clock.cc:648] HybridClock initialized: now 1785987968533283 us; error 40 us; skew 500 ppm
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:08.535885 18304 init.cc:377] Logged in from keytab as kudu/127.30.24.3@KRBTEST.COM (short username kudu)
I20260806 03:46:08.536404 18304 webserver.cc:460] Webserver started at http://127.30.24.3:33167/ using document root <none> and password file <none>
I20260806 03:46:08.536677 18304 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:08.536729 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:08.536880 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:08.537920 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "3d67af72730245a699929641926502be"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "02dabf82a94bbc55c36f3d50621c6184"
server_key_iv: "372e88001ef6f412744563e69b1316ce"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.538298 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "3d67af72730245a699929641926502be"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "02dabf82a94bbc55c36f3d50621c6184"
server_key_iv: "372e88001ef6f412744563e69b1316ce"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.539589 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:08.540335 18319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.540556 18304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:08.540658 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "3d67af72730245a699929641926502be"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "02dabf82a94bbc55c36f3d50621c6184"
server_key_iv: "372e88001ef6f412744563e69b1316ce"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.540738 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:08.576663 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:08.579144 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:08.579308 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:08.579551 18304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:08.579921 18304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:08.579977 18304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.580026 18304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:08.580053 18304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.585840 18304 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:46373
I20260806 03:46:08.585888 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:46373 every 8 connection(s)
I20260806 03:46:08.586241 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
Aug 06 03:46:08 dist-test-slave-10pc krb5kdc[17921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.3@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:08.593281 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18304
I20260806 03:46:08.593381 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:46:08.593583 30816 external_mini_cluster.cc:1428] Setting key 28f095a88361967fe945177a48364bae
I20260806 03:46:08.595736 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.587014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:39789 (local address 127.30.24.62:46447)
0806 03:46:08.587106 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.587108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.587250 (+ 142us) server_negotiation.cc:407] Connection header received
0806 03:46:08.587736 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.587737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.587748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.587801 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:08.588856 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.589347 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.590349 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.590526 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.590576 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.592766 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:08.592767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:08.592767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:08.592774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:08.594500 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.594933 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.594934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.594934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.594991 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.595219 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.595220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.595220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.595323 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:08.595375 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.595604 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.595672 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 03:46:08.595898 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.586961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:46447 (local address 127.30.24.3:39789)
0806 03:46:08.587124 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.587163 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.587647 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.587856 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.587859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.588566 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:08.588770 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.588775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.589453 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.589454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.590271 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.590274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.590596 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.590597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.590701 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.591269 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:08.591277 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:08.592674 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:08.594602 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:08.594609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:08.594612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:08.594846 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:08.595056 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:08.595057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:08.595057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:08.595163 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:08.595378 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:08.595382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:08.595497 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.595577 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.595697 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260806 03:46:08.596392 18434 heartbeater.cc:347] Connected to a master server at 127.30.24.62:46447
I20260806 03:46:08.596529 18434 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:08.596691 18434 heartbeater.cc:511] Master 127.30.24.62:46447 requested a full tablet report, sending...
I20260806 03:46:08.597048 17968 ts_manager.cc:194] Registered new tserver with Master: 3d67af72730245a699929641926502be (127.30.24.3:46373)
I20260806 03:46:08.597446 17968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.24.3@KRBTEST.COM'} at 127.30.24.3:39789
I20260806 03:46:08.605347 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:46:08.612725 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.606791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.30.24.62:46447)
0806 03:46:08.606863 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.606866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.606952 (+ 86us) server_negotiation.cc:407] Connection header received
0806 03:46:08.607030 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.607031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.607042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.607091 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:08.607673 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.608142 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.608945 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.609136 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.609211 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.610397 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:08.610398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:08.610398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:08.610406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:08.611508 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.611849 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.611850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.611851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.611911 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:08.612115 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:08.612116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:08.612116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:08.612271 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:08.612380 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.612521 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.612602 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 03:46:08.614454 17968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:46:08.616048 17968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:46:08.622615 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.618746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.30.24.2:36889)
0806 03:46:08.618936 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.618941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.618950 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:46:08.618988 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.618989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.619004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.619080 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:08.619327 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.620437 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.621709 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.622190 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.622277 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.622335 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.622475 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.622521 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260806 03:46:08.622632 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.618489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36889 (local address 127.0.0.1:35772)
0806 03:46:08.618749 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.618760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.618808 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.619192 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.619193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.619195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:08.619241 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.619242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.620547 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.620548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.621623 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.621625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.622279 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.622280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.622360 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.622374 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.622447 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.622532 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":47,"thread_start_us":79,"threads_started":1}
I20260806 03:46:08.622818 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.618511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.30.24.1:34105)
0806 03:46:08.618723 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.618728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.618739 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:08.618777 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.618779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.618794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.618891 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:08.619131 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.619998 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.621477 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.622194 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.622261 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.622318 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.622687 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.622731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260806 03:46:08.622836 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.618288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:34105 (local address 127.0.0.1:55370)
0806 03:46:08.618456 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.618496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.618558 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.618982 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.618985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.618988 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:08.619059 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.619061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.620099 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.620100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.621390 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.621393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.622355 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.622356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.622420 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.622424 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.622474 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.622547 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260806 03:46:08.623099 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.618288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:46373 (local address 127.0.0.1:55782)
0806 03:46:08.618616 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.618628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.618690 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.619387 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.619387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.619389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:08.619450 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.619451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.620706 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.620707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.621775 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.621778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.622656 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.622657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.622721 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.622725 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.622755 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.622788 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260806 03:46:08.623376 18230 tablet_service.cc:1467] Processing CreateTablet for tablet 1218d445f6474aa8abe4a06f25ecfa96 (DEFAULT_TABLE table=test-table [id=6753b432388e403890890c5358b434eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:08.623543 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.618391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.30.24.3:46373)
0806 03:46:08.619204 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.619208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.619219 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:08.619257 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.619259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.619276 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.619383 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:08.619520 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.620577 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.621870 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.622563 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.623293 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.623364 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.623415 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.623455 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":101,"threads_started":1}
I20260806 03:46:08.623797 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1218d445f6474aa8abe4a06f25ecfa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:08.623878 18095 tablet_service.cc:1467] Processing CreateTablet for tablet 1218d445f6474aa8abe4a06f25ecfa96 (DEFAULT_TABLE table=test-table [id=6753b432388e403890890c5358b434eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:08.624194 18095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1218d445f6474aa8abe4a06f25ecfa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:08.624205 18368 tablet_service.cc:1467] Processing CreateTablet for tablet 1218d445f6474aa8abe4a06f25ecfa96 (DEFAULT_TABLE table=test-table [id=6753b432388e403890890c5358b434eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:08.624536 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1218d445f6474aa8abe4a06f25ecfa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:08.626926 18451 tablet_bootstrap.cc:492] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Bootstrap starting.
I20260806 03:46:08.627606 18451 tablet_bootstrap.cc:654] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:08.627624 18452 tablet_bootstrap.cc:492] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2: Bootstrap starting.
I20260806 03:46:08.627848 18453 tablet_bootstrap.cc:492] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Bootstrap starting.
I20260806 03:46:08.628211 18451 log.cc:824] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:08.628314 18452 tablet_bootstrap.cc:654] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:08.628543 18453 tablet_bootstrap.cc:654] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:08.628882 18452 log.cc:824] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:08.629132 18453 log.cc:824] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:08.629184 18451 tablet_bootstrap.cc:492] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: No bootstrap required, opened a new log
I20260806 03:46:08.629263 18451 ts_tablet_manager.cc:1397] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 03:46:08.629627 18452 tablet_bootstrap.cc:492] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2: No bootstrap required, opened a new log
I20260806 03:46:08.629714 18452 ts_tablet_manager.cc:1397] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:46:08.629822 18453 tablet_bootstrap.cc:492] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: No bootstrap required, opened a new log
I20260806 03:46:08.629886 18453 ts_tablet_manager.cc:1397] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 03:46:08.631042 18451 raft_consensus.cc:348] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.631217 18451 raft_consensus.cc:374] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:08.631261 18451 raft_consensus.cc:729] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57688024631d48eba5fd061ec86214be, State: Initialized, Role: FOLLOWER
I20260806 03:46:08.631398 18451 consensus_queue.cc:260] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.631675 18451 ts_tablet_manager.cc:1428] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 03:46:08.631811 18299 heartbeater.cc:503] Master 127.30.24.62:46447 was elected leader, sending a full tablet report...
I20260806 03:46:08.631732 18452 raft_consensus.cc:348] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.631739 18453 raft_consensus.cc:348] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.631901 18453 raft_consensus.cc:374] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:08.631901 18452 raft_consensus.cc:374] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:08.631943 18453 raft_consensus.cc:729] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d67af72730245a699929641926502be, State: Initialized, Role: FOLLOWER
I20260806 03:46:08.631943 18452 raft_consensus.cc:729] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203dea29ff054dce9c546f4e0b3953a2, State: Initialized, Role: FOLLOWER
I20260806 03:46:08.632058 18452 consensus_queue.cc:260] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.632058 18453 consensus_queue.cc:260] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.632352 18452 ts_tablet_manager.cc:1428] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 03:46:08.632352 18453 ts_tablet_manager.cc:1428] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 03:46:08.632532 18162 heartbeater.cc:503] Master 127.30.24.62:46447 was elected leader, sending a full tablet report...
I20260806 03:46:08.632794 18434 heartbeater.cc:503] Master 127.30.24.62:46447 was elected leader, sending a full tablet report...
W20260806 03:46:08.653959 18300 tablet.cc:2369] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:08.677474 18459 raft_consensus.cc:482] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:46:08.677603 18459 raft_consensus.cc:504] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.678098 18459 leader_election.cc:288] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d67af72730245a699929641926502be (127.30.24.3:46373), 57688024631d48eba5fd061ec86214be (127.30.24.2:36889)
I20260806 03:46:08.681190 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.678401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:57723 (local address 127.30.24.3:46373)
0806 03:46:08.678469 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.678471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.678481 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:46:08.678514 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.678515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.678528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.678601 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:08.678783 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.679261 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.680363 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.680850 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.680909 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.680918 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.681090 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.681122 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 03:46:08.681192 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.678281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:46373 (local address 127.30.24.1:57723)
0806 03:46:08.678341 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.678367 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.678414 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.678608 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.678609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.678612 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:08.678692 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.678693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.679361 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.679362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.680261 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.680264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.680931 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.680932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.681015 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.681019 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.681077 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.681118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 03:46:08.681672 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.678435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:36889 (local address 127.30.24.1:44673)
0806 03:46:08.678630 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.678639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:08.678731 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:08.678878 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:08.678879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:08.678880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:08.678952 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.678953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.679667 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.679668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.680616 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:08.680619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.681465 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.681466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:08.681547 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.681551 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 03:46:08.681589 (+ 38us) client_negotiation.cc:240] Negotiation successful
0806 03:46:08.681617 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":141,"threads_started":1}
I20260806 03:46:08.681665 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218d445f6474aa8abe4a06f25ecfa96" candidate_uuid: "203dea29ff054dce9c546f4e0b3953a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d67af72730245a699929641926502be" is_pre_election: true
I20260806 03:46:08.681730 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.678498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:44673 (local address 127.30.24.2:36889)
0806 03:46:08.678543 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.678545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.678721 (+ 176us) server_negotiation.cc:407] Connection header received
0806 03:46:08.678806 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.678807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.678818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.678875 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:08.679053 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.679589 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.680719 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.681264 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.681329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.681338 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.681622 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.681669 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:46:08.681823 18388 raft_consensus.cc:2454] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 203dea29ff054dce9c546f4e0b3953a2 in term 0.
I20260806 03:46:08.682070 18049 leader_election.cc:302] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 203dea29ff054dce9c546f4e0b3953a2, 3d67af72730245a699929641926502be; no voters:
I20260806 03:46:08.682080 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218d445f6474aa8abe4a06f25ecfa96" candidate_uuid: "203dea29ff054dce9c546f4e0b3953a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57688024631d48eba5fd061ec86214be" is_pre_election: true
I20260806 03:46:08.682222 18250 raft_consensus.cc:2454] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 203dea29ff054dce9c546f4e0b3953a2 in term 0.
I20260806 03:46:08.682228 18459 raft_consensus.cc:2781] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:46:08.682330 18459 raft_consensus.cc:482] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:46:08.682365 18459 raft_consensus.cc:3037] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:08.683118 18459 raft_consensus.cc:504] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.683270 18459 leader_election.cc:288] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d67af72730245a699929641926502be (127.30.24.3:46373), 57688024631d48eba5fd061ec86214be (127.30.24.2:36889)
I20260806 03:46:08.683475 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218d445f6474aa8abe4a06f25ecfa96" candidate_uuid: "203dea29ff054dce9c546f4e0b3953a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d67af72730245a699929641926502be"
I20260806 03:46:08.683487 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218d445f6474aa8abe4a06f25ecfa96" candidate_uuid: "203dea29ff054dce9c546f4e0b3953a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57688024631d48eba5fd061ec86214be"
I20260806 03:46:08.683562 18388 raft_consensus.cc:3037] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:08.683588 18250 raft_consensus.cc:3037] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:08.684276 18388 raft_consensus.cc:2454] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 203dea29ff054dce9c546f4e0b3953a2 in term 1.
I20260806 03:46:08.684397 18250 raft_consensus.cc:2454] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 203dea29ff054dce9c546f4e0b3953a2 in term 1.
I20260806 03:46:08.684581 18049 leader_election.cc:302] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 203dea29ff054dce9c546f4e0b3953a2, 3d67af72730245a699929641926502be; no voters:
I20260806 03:46:08.684719 18459 raft_consensus.cc:2781] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:08.684921 18459 raft_consensus.cc:686] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [term 1 LEADER]: Becoming Leader. State: Replica: 203dea29ff054dce9c546f4e0b3953a2, State: Running, Role: LEADER
I20260806 03:46:08.685060 18459 consensus_queue.cc:237] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } }
I20260806 03:46:08.685953 17968 catalog_manager.cc:5495] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 203dea29ff054dce9c546f4e0b3953a2 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "203dea29ff054dce9c546f4e0b3953a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203dea29ff054dce9c546f4e0b3953a2" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 34105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 } health_report { overall_health: UNKNOWN } } }
I20260806 03:46:08.699868 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.697704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.30.24.1:34105)
0806 03:46:08.697744 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.697746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.697753 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:46:08.697831 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.697831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.697849 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.697916 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:08.698052 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.698491 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.699206 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.699385 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.699435 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.699560 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:08.699658 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:08.699719 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.699751 (+ 32us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.699767 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 03:46:08.701783 18388 raft_consensus.cc:1261] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 1 FOLLOWER]: Refusing update from remote peer 203dea29ff054dce9c546f4e0b3953a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:08.701817 18250 raft_consensus.cc:1261] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 1 FOLLOWER]: Refusing update from remote peer 203dea29ff054dce9c546f4e0b3953a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:08.702167 18459 consensus_queue.cc:1035] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d67af72730245a699929641926502be" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:08.702317 18462 consensus_queue.cc:1035] T 1218d445f6474aa8abe4a06f25ecfa96 P 203dea29ff054dce9c546f4e0b3953a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57688024631d48eba5fd061ec86214be" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:08.707072 17968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:08.707186 17968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:08.707902 17968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99b4c3f039a4829a86e9f9bb287f7f9: Sending DeleteTablet for 3 replicas of tablet 1218d445f6474aa8abe4a06f25ecfa96
I20260806 03:46:08.708252 18095 tablet_service.cc:1514] Processing DeleteTablet for tablet 1218d445f6474aa8abe4a06f25ecfa96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:08 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:55370
I20260806 03:46:08.708357 18368 tablet_service.cc:1514] Processing DeleteTablet for tablet 1218d445f6474aa8abe4a06f25ecfa96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:08 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:55782
I20260806 03:46:08.708359 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet 1218d445f6474aa8abe4a06f25ecfa96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:08 UTC) from {username='kudu', principal='kudu/127.30.24.62@KRBTEST.COM'} at 127.0.0.1:35772
I20260806 03:46:08.708583 18476 tablet_replica.cc:330] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: stopping tablet replica
I20260806 03:46:08.708693 18476 raft_consensus.cc:2229] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:08.708793 18476 raft_consensus.cc:2258] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:08.708823 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18030
I20260806 03:46:08.709877 18476 ts_tablet_manager.cc:1905] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:08.710615 18477 tablet_replica.cc:330] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: stopping tablet replica
I20260806 03:46:08.710740 18477 raft_consensus.cc:2229] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:08.710839 18477 raft_consensus.cc:2258] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:08.711246 18477 ts_tablet_manager.cc:1905] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:08.711891 18476 ts_tablet_manager.cc:1918] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:08.711973 18476 log.cc:1196] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/1218d445f6474aa8abe4a06f25ecfa96
I20260806 03:46:08.712288 18476 ts_tablet_manager.cc:1939] T 1218d445f6474aa8abe4a06f25ecfa96 P 3d67af72730245a699929641926502be: Deleting consensus metadata
I20260806 03:46:08.712733 18477 ts_tablet_manager.cc:1918] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:08.712808 18477 log.cc:1196] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/1218d445f6474aa8abe4a06f25ecfa96
I20260806 03:46:08.712795 17954 catalog_manager.cc:4841] TS 3d67af72730245a699929641926502be (127.30.24.3:46373): tablet 1218d445f6474aa8abe4a06f25ecfa96 (table test-table [id=6753b432388e403890890c5358b434eb]) successfully deleted
I20260806 03:46:08.713064 18477 ts_tablet_manager.cc:1939] T 1218d445f6474aa8abe4a06f25ecfa96 P 57688024631d48eba5fd061ec86214be: Deleting consensus metadata
I20260806 03:46:08.713454 17954 catalog_manager.cc:4841] TS 57688024631d48eba5fd061ec86214be (127.30.24.2:36889): tablet 1218d445f6474aa8abe4a06f25ecfa96 (table test-table [id=6753b432388e403890890c5358b434eb]) successfully deleted
W20260806 03:46:08.716879 17951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:34105 (error 108)
I20260806 03:46:08.717248 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18167
I20260806 03:46:08.717404 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.717245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:34105 (local address 127.0.0.1:55400)
0806 03:46:08.717320 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:08.717366 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 03:46:08.718269 17951 catalog_manager.cc:4584] TS 203dea29ff054dce9c546f4e0b3953a2 (127.30.24.1:34105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1218d445f6474aa8abe4a06f25ecfa96: Network error: Client connection negotiation failed: client connection to 127.30.24.1:34105: connect: Connection refused (error 111)
I20260806 03:46:08.722997 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18304
I20260806 03:46:08.728232 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 17934
2026-08-06T03:46:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2024 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T03:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:08Z Disabled control of system clock
I20260806 03:46:08.757733 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35845
--webserver_interface=127.30.24.62
--webserver_port=0
--builtin_ntp_servers=127.30.24.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:08.841153 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:08.841383 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:08.841404 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:08.842654 18482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:08.842711 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:08.842724 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:08.842736 18482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:08.842787 18482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:08.844161 18482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:35845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:08.844427 18482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:08.844782 18482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:46:08.849750 18482 server_base.cc:1034] running on GCE node
W20260806 03:46:08.849731 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:08.849723 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:08.849746 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:08.850354 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:08.850692 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:08.851872 18482 hybrid_clock.cc:648] HybridClock initialized: now 1785987968851864 us; error 36 us; skew 500 ppm
I20260806 03:46:08.853291 18482 webserver.cc:460] Webserver started at http://127.30.24.62:45315/ using document root <none> and password file <none>
I20260806 03:46:08.853564 18482 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:08.853641 18482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:08.853798 18482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:08.854909 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "cb79ba89bcb7454f91de4cc1b55a3d74"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "09781ff660e3e115542fb9381ee59334"
server_key_iv: "04159a8607d494b20ecc09f1c00967b1"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.855291 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "cb79ba89bcb7454f91de4cc1b55a3d74"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "09781ff660e3e115542fb9381ee59334"
server_key_iv: "04159a8607d494b20ecc09f1c00967b1"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.856669 18482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:08.857396 18497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.857573 18482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:08.857674 18482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "cb79ba89bcb7454f91de4cc1b55a3d74"
format_stamp: "Formatted at 2026-08-06 03:46:08 on dist-test-slave-10pc"
server_key: "09781ff660e3e115542fb9381ee59334"
server_key_iv: "04159a8607d494b20ecc09f1c00967b1"
server_key_version: "encryptionkey@0"
I20260806 03:46:08.857760 18482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:08.882308 18482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:08.882683 18482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:08.882853 18482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:08.886992 18482 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:35845
I20260806 03:46:08.887074 18549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:35845 every 8 connection(s)
I20260806 03:46:08.887422 18482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:08.888528 18550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:08.890905 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74: Bootstrap starting.
I20260806 03:46:08.891535 18550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:08.891841 18550 log.cc:824] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:08.892678 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74: No bootstrap required, opened a new log
I20260806 03:46:08.892776 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18482
I20260806 03:46:08.892895 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:08.893092 30816 external_mini_cluster.cc:1428] Setting key 235235dc4ac9cb3f7e05931234cfb91e
I20260806 03:46:08.894799 18550 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35845 } }
I20260806 03:46:08.894941 18550 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:08.894963 18550 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb79ba89bcb7454f91de4cc1b55a3d74, State: Initialized, Role: FOLLOWER
I20260806 03:46:08.895153 18550 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35845 } }
I20260806 03:46:08.895255 18550 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:08.895306 18550 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:08.895370 18550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:08.895967 18550 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35845 } }
I20260806 03:46:08.896095 18550 leader_election.cc:302] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb79ba89bcb7454f91de4cc1b55a3d74; no voters:
I20260806 03:46:08.896289 18550 leader_election.cc:288] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:08.896337 18555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:08.896539 18555 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [term 1 LEADER]: Becoming Leader. State: Replica: cb79ba89bcb7454f91de4cc1b55a3d74, State: Running, Role: LEADER
I20260806 03:46:08.896627 18550 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:08.896629 18555 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35845 } }
I20260806 03:46:08.896968 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb79ba89bcb7454f91de4cc1b55a3d74. Latest consensus state: current_term: 1 leader_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35845 } } }
I20260806 03:46:08.897063 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:08.896979 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb79ba89bcb7454f91de4cc1b55a3d74" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35845 } } }
I20260806 03:46:08.897248 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:08.897313 18561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:08.897831 18561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:08.899753 18561 catalog_manager.cc:1347] Generated new cluster ID: a94f448f5b3c499c943ca8b68ebfbbeb
I20260806 03:46:08.899816 18561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:08.900576 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:08.893687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.30.24.62:35845)
0806 03:46:08.893892 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:08.893900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:08.893926 (+ 26us) server_negotiation.cc:407] Connection header received
0806 03:46:08.894790 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:08.894799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:08.894866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:08.895046 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:08.895292 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.897006 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.898257 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:08.898877 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:08.899030 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:08.899823 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:08.899827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:08.899827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:08.900030 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:08.900065 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:08.900077 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:08.900156 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:08.900232 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 03:46:08.900311 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260806 03:46:08.923347 18561 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:08.923920 18561 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:08.930106 18561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74: Generated new TSK 0
I20260806 03:46:08.930313 18561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:08.934434 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.1:0
--local_ip_for_outbound_sockets=127.30.24.1
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35845
--builtin_ntp_servers=127.30.24.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:09.017555 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:09.017750 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:09.017771 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:09.018908 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:09.018947 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:09.018971 18575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.1
I20260806 03:46:09.020319 18575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.24.1
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.1
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:09.020594 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:09.020907 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:09.025934 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:09.026057 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:09.025965 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:09.026325 18575 server_base.cc:1034] running on GCE node
I20260806 03:46:09.026554 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:09.026790 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:09.027956 18575 hybrid_clock.cc:648] HybridClock initialized: now 1785987969027940 us; error 34 us; skew 500 ppm
I20260806 03:46:09.029374 18575 webserver.cc:460] Webserver started at http://127.30.24.1:34897/ using document root <none> and password file <none>
I20260806 03:46:09.029637 18575 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:09.029713 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:09.029878 18575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:09.030962 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data/instance:
uuid: "947b3aae23ad4a7986f6eb72cdcc2504"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "e45fcbe66d3b07dc6aab03cac80a9950"
server_key_iv: "fe15152719a71dd18f2a578dbfa20bf7"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.031329 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance:
uuid: "947b3aae23ad4a7986f6eb72cdcc2504"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "e45fcbe66d3b07dc6aab03cac80a9950"
server_key_iv: "fe15152719a71dd18f2a578dbfa20bf7"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.032776 18575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:09.033644 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.033850 18575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:09.033957 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal
uuid: "947b3aae23ad4a7986f6eb72cdcc2504"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "e45fcbe66d3b07dc6aab03cac80a9950"
server_key_iv: "fe15152719a71dd18f2a578dbfa20bf7"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.034032 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:09.052881 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:09.053200 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:09.053349 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:09.053601 18575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:09.053936 18575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:09.053990 18575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.054031 18575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:09.054059 18575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.059937 18575 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.1:36859
I20260806 03:46:09.060000 18703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.1:36859 every 8 connection(s)
I20260806 03:46:09.060369 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/data/info.pb
I20260806 03:46:09.065950 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.061207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:56389 (local address 127.30.24.62:35845)
0806 03:46:09.061280 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.061282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.061485 (+ 203us) server_negotiation.cc:407] Connection header received
0806 03:46:09.062067 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.062068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.062079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.062165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:09.062860 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.063365 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.064328 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.064546 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.064611 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.065745 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:09.065746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:09.065746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:09.065750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:09.065764 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:09.065768 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:09.065813 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.065863 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.065888 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 03:46:09.066250 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.061171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35845 (local address 127.30.24.1:56389)
0806 03:46:09.061351 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.061393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.061967 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.062261 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.062265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.062274 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:09.062779 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.062783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.063446 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.063447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.064232 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.064233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.064780 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.064782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.064918 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.065594 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:09.065595 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:09.065625 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:09.065650 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:09.065814 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:09.065817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:09.065819 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.065865 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.065944 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260806 03:46:09.066833 18704 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35845
I20260806 03:46:09.066921 18704 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:09.067075 18704 heartbeater.cc:511] Master 127.30.24.62:35845 requested a full tablet report, sending...
I20260806 03:46:09.067607 18514 ts_manager.cc:194] Registered new tserver with Master: 947b3aae23ad4a7986f6eb72cdcc2504 (127.30.24.1:36859)
I20260806 03:46:09.068192 18514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.1:56389
I20260806 03:46:09.070396 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18575
I20260806 03:46:09.070487 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-0/wal/instance
I20260806 03:46:09.070623 30816 external_mini_cluster.cc:1428] Setting key ce75e1cc47112df6408129e0e220b37a
I20260806 03:46:09.071488 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.2:0
--local_ip_for_outbound_sockets=127.30.24.2
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35845
--builtin_ntp_servers=127.30.24.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:46:09.155364 18707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:09.155607 18707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:09.155647 18707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:09.156889 18707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:09.156951 18707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:09.156993 18707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.2
I20260806 03:46:09.158380 18707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.24.2
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.2
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:09.158644 18707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:09.158965 18707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:09.164301 18713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:09.164301 18715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:09.164580 18707 server_base.cc:1034] running on GCE node
W20260806 03:46:09.164333 18712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:09.164891 18707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:09.165167 18707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:09.166316 18707 hybrid_clock.cc:648] HybridClock initialized: now 1785987969166301 us; error 50 us; skew 500 ppm
I20260806 03:46:09.167745 18707 webserver.cc:460] Webserver started at http://127.30.24.2:33033/ using document root <none> and password file <none>
I20260806 03:46:09.168062 18707 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:09.168185 18707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:09.168358 18707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:09.169705 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data/instance:
uuid: "a4e4e4626bbe49da8c22839abbf8b468"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "c54bfa37aa45b77c78895f9a18fc97ba"
server_key_iv: "273b0c2d43cb5b64b7b6cd0c3bbefd2a"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.170142 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance:
uuid: "a4e4e4626bbe49da8c22839abbf8b468"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "c54bfa37aa45b77c78895f9a18fc97ba"
server_key_iv: "273b0c2d43cb5b64b7b6cd0c3bbefd2a"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.171665 18707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:46:09.172550 18721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.172729 18707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:09.172833 18707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal
uuid: "a4e4e4626bbe49da8c22839abbf8b468"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "c54bfa37aa45b77c78895f9a18fc97ba"
server_key_iv: "273b0c2d43cb5b64b7b6cd0c3bbefd2a"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.172919 18707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:09.185020 18707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:09.185367 18707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:09.185525 18707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:09.185786 18707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:09.186138 18707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:09.186203 18707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.186245 18707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:09.186271 18707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.192039 18707 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.2:37569
I20260806 03:46:09.192112 18835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.2:37569 every 8 connection(s)
I20260806 03:46:09.192538 18707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/data/info.pb
I20260806 03:46:09.196806 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18707
I20260806 03:46:09.196930 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal/instance
I20260806 03:46:09.197156 30816 external_mini_cluster.cc:1428] Setting key ef61d01d806f9d5652a375b032d6bd90
I20260806 03:46:09.198240 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.24.3:0
--local_ip_for_outbound_sockets=127.30.24.3
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35845
--builtin_ntp_servers=127.30.24.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:46:09.198318 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.193370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.2:36751 (local address 127.30.24.62:35845)
0806 03:46:09.193445 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.193447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.193712 (+ 265us) server_negotiation.cc:407] Connection header received
0806 03:46:09.194198 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.194199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.194209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.194263 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:09.194916 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.195406 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.196469 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.196832 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.196912 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.198044 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:09.198045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:09.198046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:09.198051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:09.198065 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:09.198069 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:09.198115 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.198208 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.198238 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:09.198537 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.193365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35845 (local address 127.30.24.2:36751)
0806 03:46:09.193586 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.193620 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.194111 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.194319 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.194322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.194330 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:09.194847 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.194851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.195486 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.195487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.196374 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.196377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.196924 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.196925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.197185 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.197895 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:09.197895 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:09.197936 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:09.197967 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:09.198115 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:09.198120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:09.198123 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.198201 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.198306 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260806 03:46:09.199008 18836 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35845
I20260806 03:46:09.199100 18836 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:09.199307 18836 heartbeater.cc:511] Master 127.30.24.62:35845 requested a full tablet report, sending...
I20260806 03:46:09.199710 18514 ts_manager.cc:194] Registered new tserver with Master: a4e4e4626bbe49da8c22839abbf8b468 (127.30.24.2:37569)
I20260806 03:46:09.200178 18514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.2:36751
W20260806 03:46:09.285377 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:09.285583 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:09.285604 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:09.286772 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:09.286818 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:09.286841 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.24.3
I20260806 03:46:09.288410 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.24.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.24.3
--webserver_port=0
--tserver_master_addrs=127.30.24.62:35845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.24.3
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:09.288718 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:09.289034 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:09.294445 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:09.294479 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:09.294498 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:09.294690 18839 server_base.cc:1034] running on GCE node
I20260806 03:46:09.294888 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:09.295133 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:09.296295 18839 hybrid_clock.cc:648] HybridClock initialized: now 1785987969296271 us; error 42 us; skew 500 ppm
I20260806 03:46:09.297767 18839 webserver.cc:460] Webserver started at http://127.30.24.3:34195/ using document root <none> and password file <none>
I20260806 03:46:09.298086 18839 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:09.298135 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:09.298309 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:09.299398 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data/instance:
uuid: "dc7a876872ac47b4834cb1123b24b90b"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "77c8185723ffd35c25f36281a63dd225"
server_key_iv: "4327b182846f0200351b41881983f780"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.299767 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance:
uuid: "dc7a876872ac47b4834cb1123b24b90b"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "77c8185723ffd35c25f36281a63dd225"
server_key_iv: "4327b182846f0200351b41881983f780"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.301141 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260806 03:46:09.301996 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.302258 18839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:09.302357 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal
uuid: "dc7a876872ac47b4834cb1123b24b90b"
format_stamp: "Formatted at 2026-08-06 03:46:09 on dist-test-slave-10pc"
server_key: "77c8185723ffd35c25f36281a63dd225"
server_key_iv: "4327b182846f0200351b41881983f780"
server_key_version: "encryptionkey@0"
I20260806 03:46:09.302448 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:09.318485 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:09.318827 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:09.318994 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:09.319243 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:46:09.319587 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:46:09.319643 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.319681 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:46:09.319727 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:09.325585 18839 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.3:43681
I20260806 03:46:09.325636 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.3:43681 every 8 connection(s)
I20260806 03:46:09.325975 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/data/info.pb
I20260806 03:46:09.331884 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.326800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.3:44497 (local address 127.30.24.62:35845)
0806 03:46:09.326874 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.326877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.327857 (+ 980us) server_negotiation.cc:407] Connection header received
0806 03:46:09.328349 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.328350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.328361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.328420 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:09.329168 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.329652 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.330609 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.330797 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.330848 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.331622 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:09.331623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:09.331623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:09.331627 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:09.331642 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:09.331646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:09.331700 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.331793 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.331822 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:46:09.332068 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.326719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:35845 (local address 127.30.24.3:44497)
0806 03:46:09.327735 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.327771 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.328263 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.328530 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.328534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.328545 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:09.329081 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.329085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.329743 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.329744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.330531 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.330533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.330871 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.330872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.330982 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.331484 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:46:09.331484 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:09.331514 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:46:09.331538 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:09.331714 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:09.331718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:09.331721 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.331778 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.331859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":89,"threads_started":1}
I20260806 03:46:09.332619 18969 heartbeater.cc:347] Connected to a master server at 127.30.24.62:35845
I20260806 03:46:09.332733 18969 heartbeater.cc:464] Registering TS with master...
I20260806 03:46:09.332918 18969 heartbeater.cc:511] Master 127.30.24.62:35845 requested a full tablet report, sending...
I20260806 03:46:09.333323 18514 ts_manager.cc:194] Registered new tserver with Master: dc7a876872ac47b4834cb1123b24b90b (127.30.24.3:43681)
I20260806 03:46:09.333704 18514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.24.3:44497
I20260806 03:46:09.334455 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 18839
I20260806 03:46:09.334581 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal/instance
I20260806 03:46:09.334736 30816 external_mini_cluster.cc:1428] Setting key 5de2327d09d5f9760fd948ab8c17f80f
I20260806 03:46:09.335449 30816 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:46:09.339924 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.336885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.30.24.62:35845)
0806 03:46:09.336944 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.336946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.337178 (+ 232us) server_negotiation.cc:407] Connection header received
0806 03:46:09.337210 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.337211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.337249 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.337302 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:09.337496 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.338003 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.338796 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.338978 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.339070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.339671 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:09.339673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:09.339673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:09.339677 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:09.339691 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:09.339695 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:09.339740 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.339787 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.339810 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 03:46:09.341658 18514 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:46:09.343286 18514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:46:09.349990 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.345720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:43681 (local address 127.0.0.1:52370)
0806 03:46:09.346372 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.346383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.346467 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.346767 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.346768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.346769 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:09.346803 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.346805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.348039 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.348040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.348971 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.348974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.349680 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.349758 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.349772 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.349851 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.349907 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":103,"threads_started":1}
I20260806 03:46:09.350006 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.345998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.30.24.3:43681)
0806 03:46:09.346218 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.346223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.346437 (+ 214us) server_negotiation.cc:407] Connection header received
0806 03:46:09.346593 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.346596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.346613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.346699 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:09.346886 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.347950 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349058 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.349605 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.349779 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.349866 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.349906 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260806 03:46:09.350391 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.346130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.30.24.1:36859)
0806 03:46:09.346355 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.346359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.346370 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:09.346411 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.346413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.346429 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.346618 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:09.346832 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.347551 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349130 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.349885 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349958 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.350040 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.350255 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.350300 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":117,"threads_started":1}
I20260806 03:46:09.350543 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.345888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.30.24.2:37569)
0806 03:46:09.346094 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.346100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.346219 (+ 119us) server_negotiation.cc:407] Connection header received
0806 03:46:09.346276 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.346278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.346294 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.346384 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:09.346636 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.347791 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349131 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.349885 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.349958 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.350040 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.350401 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.350447 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":129,"threads_started":1}
I20260806 03:46:09.350360 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.345702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:36859 (local address 127.0.0.1:47162)
0806 03:46:09.345903 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.345947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.346018 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.346698 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.346699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.346700 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:09.346748 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.346750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.347649 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.347651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.349022 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.349025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.350081 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.350082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.350154 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.350160 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.350216 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.350264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":29,"thread_start_us":86,"threads_started":1}
I20260806 03:46:09.350360 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.345703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:37569 (local address 127.0.0.1:53318)
0806 03:46:09.346141 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.346151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.346204 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.346467 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.346471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.346473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:09.346550 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.346552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.347883 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.347884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.349022 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.349025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.350084 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.350085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.350154 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.350175 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.350222 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.350266 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":15,"spinlock_wait_cycles":768,"thread_start_us":123,"threads_started":1}
I20260806 03:46:09.351326 18901 tablet_service.cc:1467] Processing CreateTablet for tablet 66724020492241e596cdde753e66ec8e (DEFAULT_TABLE table=test-table [id=d16c79e07204492aa79fb7c161dd6737]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:09.351326 18637 tablet_service.cc:1467] Processing CreateTablet for tablet 66724020492241e596cdde753e66ec8e (DEFAULT_TABLE table=test-table [id=d16c79e07204492aa79fb7c161dd6737]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:09.351645 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66724020492241e596cdde753e66ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:09.351644 18901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66724020492241e596cdde753e66ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:09.352972 18769 tablet_service.cc:1467] Processing CreateTablet for tablet 66724020492241e596cdde753e66ec8e (DEFAULT_TABLE table=test-table [id=d16c79e07204492aa79fb7c161dd6737]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:46:09.353371 18769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66724020492241e596cdde753e66ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:09.354413 18983 tablet_bootstrap.cc:492] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: Bootstrap starting.
I20260806 03:46:09.355087 18983 tablet_bootstrap.cc:654] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:09.355332 18983 log.cc:824] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:09.355705 18984 tablet_bootstrap.cc:492] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Bootstrap starting.
I20260806 03:46:09.356055 18983 tablet_bootstrap.cc:492] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: No bootstrap required, opened a new log
I20260806 03:46:09.356103 18983 ts_tablet_manager.cc:1397] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:46:09.356449 18986 tablet_bootstrap.cc:492] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Bootstrap starting.
I20260806 03:46:09.357214 18986 tablet_bootstrap.cc:654] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:09.357540 18986 log.cc:824] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:09.358342 18986 tablet_bootstrap.cc:492] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: No bootstrap required, opened a new log
I20260806 03:46:09.358418 18986 ts_tablet_manager.cc:1397] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 03:46:09.358906 18983 raft_consensus.cc:348] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.359118 18983 raft_consensus.cc:374] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:09.359166 18983 raft_consensus.cc:729] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947b3aae23ad4a7986f6eb72cdcc2504, State: Initialized, Role: FOLLOWER
I20260806 03:46:09.359316 18983 consensus_queue.cc:260] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.359604 18983 ts_tablet_manager.cc:1428] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 03:46:09.359670 18704 heartbeater.cc:503] Master 127.30.24.62:35845 was elected leader, sending a full tablet report...
I20260806 03:46:09.360941 18984 tablet_bootstrap.cc:654] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:09.361296 18984 log.cc:824] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:09.361197 18986 raft_consensus.cc:348] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.361361 18986 raft_consensus.cc:374] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:09.361394 18986 raft_consensus.cc:729] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e4e4626bbe49da8c22839abbf8b468, State: Initialized, Role: FOLLOWER
I20260806 03:46:09.361519 18986 consensus_queue.cc:260] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.361826 18986 ts_tablet_manager.cc:1428] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 03:46:09.361860 18836 heartbeater.cc:503] Master 127.30.24.62:35845 was elected leader, sending a full tablet report...
I20260806 03:46:09.362115 18984 tablet_bootstrap.cc:492] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: No bootstrap required, opened a new log
I20260806 03:46:09.362207 18984 ts_tablet_manager.cc:1397] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260806 03:46:09.364236 18984 raft_consensus.cc:348] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.364411 18984 raft_consensus.cc:374] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:09.364480 18984 raft_consensus.cc:729] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7a876872ac47b4834cb1123b24b90b, State: Initialized, Role: FOLLOWER
I20260806 03:46:09.364598 18984 consensus_queue.cc:260] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.364897 18984 ts_tablet_manager.cc:1428] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 03:46:09.364934 18969 heartbeater.cc:503] Master 127.30.24.62:35845 was elected leader, sending a full tablet report...
W20260806 03:46:09.443137 18837 tablet.cc:2369] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:46:09.561157 18705 tablet.cc:2369] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:46:09.576625 18970 tablet.cc:2369] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:46:09.628756 18988 raft_consensus.cc:482] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:46:09.628904 18988 raft_consensus.cc:504] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.629303 18988 leader_election.cc:288] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e4e4626bbe49da8c22839abbf8b468 (127.30.24.2:37569), dc7a876872ac47b4834cb1123b24b90b (127.30.24.3:43681)
I20260806 03:46:09.633392 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.629554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.3:43681 (local address 127.30.24.1:34315)
0806 03:46:09.629655 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.629667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.629730 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.629896 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.629896 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.629898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:09.629967 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.629969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.630907 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.630908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.632209 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.632213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.633077 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.633078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.633186 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.633191 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.633243 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.633289 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260806 03:46:09.633375 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.629698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:34315 (local address 127.30.24.3:43681)
0806 03:46:09.629763 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.629765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.629780 (+ 15us) server_negotiation.cc:407] Connection header received
0806 03:46:09.629811 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.629813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.629820 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.629875 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:09.630075 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.630790 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.632328 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.632936 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.632992 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.633006 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.633266 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.633309 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 03:46:09.633797 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66724020492241e596cdde753e66ec8e" candidate_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7a876872ac47b4834cb1123b24b90b" is_pre_election: true
I20260806 03:46:09.634002 18921 raft_consensus.cc:2454] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 947b3aae23ad4a7986f6eb72cdcc2504 in term 0.
I20260806 03:46:09.634275 18590 leader_election.cc:302] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 947b3aae23ad4a7986f6eb72cdcc2504, dc7a876872ac47b4834cb1123b24b90b; no voters:
I20260806 03:46:09.634428 18988 raft_consensus.cc:2781] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:46:09.634512 18988 raft_consensus.cc:482] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:46:09.634538 18988 raft_consensus.cc:3037] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:09.635020 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.629519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.2:37569 (local address 127.30.24.1:36483)
0806 03:46:09.629644 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.629667 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:09.629731 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:09.630021 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:09.630022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:09.630023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:46:09.630082 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.630084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.631244 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.631245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.632281 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:09.632285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.634755 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.634756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:09.634823 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.634828 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 03:46:09.634886 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 03:46:09.634926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260806 03:46:09.635493 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.629573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.24.1:36483 (local address 127.30.24.2:37569)
0806 03:46:09.629627 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.629629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.629912 (+ 283us) server_negotiation.cc:407] Connection header received
0806 03:46:09.629942 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.629943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.629951 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.629990 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:46:09.630171 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.630861 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.632408 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.634620 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.634693 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.634705 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.635344 (+ 639us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.635408 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 03:46:09.635600 18988 raft_consensus.cc:504] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.635764 18988 leader_election.cc:288] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [CANDIDATE]: Term 1 election: Requested vote from peers a4e4e4626bbe49da8c22839abbf8b468 (127.30.24.2:37569), dc7a876872ac47b4834cb1123b24b90b (127.30.24.3:43681)
I20260806 03:46:09.636053 18789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66724020492241e596cdde753e66ec8e" candidate_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e4e4626bbe49da8c22839abbf8b468" is_pre_election: true
I20260806 03:46:09.636121 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66724020492241e596cdde753e66ec8e" candidate_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7a876872ac47b4834cb1123b24b90b"
I20260806 03:46:09.636234 18789 raft_consensus.cc:2454] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 947b3aae23ad4a7986f6eb72cdcc2504 in term 0.
I20260806 03:46:09.636200 18921 raft_consensus.cc:3037] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:09.636059 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66724020492241e596cdde753e66ec8e" candidate_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e4e4626bbe49da8c22839abbf8b468"
I20260806 03:46:09.636385 18788 raft_consensus.cc:3037] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:09.637225 18921 raft_consensus.cc:2454] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 947b3aae23ad4a7986f6eb72cdcc2504 in term 1.
I20260806 03:46:09.637323 18788 raft_consensus.cc:2454] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 947b3aae23ad4a7986f6eb72cdcc2504 in term 1.
I20260806 03:46:09.637457 18590 leader_election.cc:302] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 947b3aae23ad4a7986f6eb72cdcc2504, dc7a876872ac47b4834cb1123b24b90b; no voters:
I20260806 03:46:09.637661 18988 raft_consensus.cc:2781] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:09.637862 18988 raft_consensus.cc:686] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [term 1 LEADER]: Becoming Leader. State: Replica: 947b3aae23ad4a7986f6eb72cdcc2504, State: Running, Role: LEADER
I20260806 03:46:09.637964 18988 consensus_queue.cc:237] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } }
I20260806 03:46:09.638991 18514 catalog_manager.cc:5495] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 947b3aae23ad4a7986f6eb72cdcc2504 (127.30.24.1). New cstate: current_term: 1 leader_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "947b3aae23ad4a7986f6eb72cdcc2504" member_type: VOTER last_known_addr { host: "127.30.24.1" port: 36859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 } health_report { overall_health: UNKNOWN } } }
I20260806 03:46:09.706050 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.703527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.30.24.1:36859)
0806 03:46:09.703602 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:09.703605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:09.703635 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:46:09.703693 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:09.703694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:09.703716 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:09.703792 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:09.703945 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.704429 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.705295 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:09.705483 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:09.705537 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:09.705680 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:09.705789 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:09.705908 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:09.705922 (+ 14us) server_negotiation.cc:299] Negotiation successful
0806 03:46:09.705942 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 03:46:09.708353 18788 raft_consensus.cc:1261] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 1 FOLLOWER]: Refusing update from remote peer 947b3aae23ad4a7986f6eb72cdcc2504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:09.708353 18921 raft_consensus.cc:1261] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 1 FOLLOWER]: Refusing update from remote peer 947b3aae23ad4a7986f6eb72cdcc2504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:09.708837 18988 consensus_queue.cc:1035] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc7a876872ac47b4834cb1123b24b90b" member_type: VOTER last_known_addr { host: "127.30.24.3" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:09.709076 18994 consensus_queue.cc:1035] T 66724020492241e596cdde753e66ec8e P 947b3aae23ad4a7986f6eb72cdcc2504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e4e4626bbe49da8c22839abbf8b468" member_type: VOTER last_known_addr { host: "127.30.24.2" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:09.711592 18997 mvcc.cc:205] Tried to move back new op lower bound from 7315406723921698816 to 7315406723637841920. Current Snapshot: MvccSnapshot[applied={T|T < 7315406723921698816}]
I20260806 03:46:09.714169 18514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:09.714255 18514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:46:09.715039 18514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb79ba89bcb7454f91de4cc1b55a3d74: Sending DeleteTablet for 3 replicas of tablet 66724020492241e596cdde753e66ec8e
I20260806 03:46:09.715369 18769 tablet_service.cc:1514] Processing DeleteTablet for tablet 66724020492241e596cdde753e66ec8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:09 UTC) from {username='slave'} at 127.0.0.1:53318
I20260806 03:46:09.715435 18637 tablet_service.cc:1514] Processing DeleteTablet for tablet 66724020492241e596cdde753e66ec8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:09 UTC) from {username='slave'} at 127.0.0.1:47162
I20260806 03:46:09.715495 18901 tablet_service.cc:1514] Processing DeleteTablet for tablet 66724020492241e596cdde753e66ec8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:46:09 UTC) from {username='slave'} at 127.0.0.1:52370
I20260806 03:46:09.715628 19007 tablet_replica.cc:330] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: stopping tablet replica
I20260806 03:46:09.715770 19007 raft_consensus.cc:2229] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:09.715863 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18575
I20260806 03:46:09.716975 19009 tablet_replica.cc:330] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: stopping tablet replica
I20260806 03:46:09.717090 19009 raft_consensus.cc:2229] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:46:09.717209 19009 raft_consensus.cc:2258] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:09.717537 19007 raft_consensus.cc:2258] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:46:09.718019 19007 ts_tablet_manager.cc:1905] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:09.718295 19009 ts_tablet_manager.cc:1905] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:46:09.719794 19007 ts_tablet_manager.cc:1918] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:09.719878 19007 log.cc:1196] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-1/wal/wals/66724020492241e596cdde753e66ec8e
I20260806 03:46:09.720098 19009 ts_tablet_manager.cc:1918] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:46:09.720172 19007 ts_tablet_manager.cc:1939] T 66724020492241e596cdde753e66ec8e P a4e4e4626bbe49da8c22839abbf8b468: Deleting consensus metadata
I20260806 03:46:09.720247 19009 log.cc:1196] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Deleting WAL directory at /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785987724420712-30816-0/minicluster-data/ts-2/wal/wals/66724020492241e596cdde753e66ec8e
I20260806 03:46:09.720566 19009 ts_tablet_manager.cc:1939] T 66724020492241e596cdde753e66ec8e P dc7a876872ac47b4834cb1123b24b90b: Deleting consensus metadata
I20260806 03:46:09.721001 18498 catalog_manager.cc:4841] TS dc7a876872ac47b4834cb1123b24b90b (127.30.24.3:43681): tablet 66724020492241e596cdde753e66ec8e (table test-table [id=d16c79e07204492aa79fb7c161dd6737]) successfully deleted
I20260806 03:46:09.721104 18501 catalog_manager.cc:4841] TS a4e4e4626bbe49da8c22839abbf8b468 (127.30.24.2:37569): tablet 66724020492241e596cdde753e66ec8e (table test-table [id=d16c79e07204492aa79fb7c161dd6737]) successfully deleted
W20260806 03:46:09.725446 18500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.24.1:36859 (error 108)
I20260806 03:46:09.725656 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18707
I20260806 03:46:09.728572 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:09.728374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.1:36859 (local address 127.0.0.1:47182)
0806 03:46:09.728447 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:09.728531 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.1:36859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 03:46:09.728726 18500 catalog_manager.cc:4584] TS 947b3aae23ad4a7986f6eb72cdcc2504 (127.30.24.1:36859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66724020492241e596cdde753e66ec8e: Network error: Client connection negotiation failed: client connection to 127.30.24.1:36859: connect: Connection refused (error 111)
I20260806 03:46:09.732262 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18839
I20260806 03:46:09.737566 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 18482
2026-08-06T03:46:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1008 ms)
[----------] 6 tests from EncryptionPolicyTest (9209 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:09 dist-test-slave-10pc krb5kdc[19013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:09 dist-test-slave-10pc krb5kdc[19013](info): set up 2 sockets
Aug 06 03:46:09 dist-test-slave-10pc krb5kdc[19013](info): commencing operation
krb5kdc: starting...
W20260806 03:46:10.821771 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:10 dist-test-slave-10pc krb5kdc[19013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:10.953575 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:11.038090 19026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:11.038357 19026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:11.038396 19026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:11.039551 19026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:11.039642 19026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:11.039671 19026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:11.039702 19026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:11.039724 19026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:11.041292 19026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:11.041591 19026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:11.041834 19026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:46:11.046964 19026 server_base.cc:1034] running on GCE node
W20260806 03:46:11.046906 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:11.046988 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:11.046945 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:11.047358 19026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:11.047680 19026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:11.048847 19026 hybrid_clock.cc:648] HybridClock initialized: now 1785987971048835 us; error 30 us; skew 500 ppm
Aug 06 03:46:11 dist-test-slave-10pc krb5kdc[19013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:11.056592 19026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:11.057114 19026 webserver.cc:460] Webserver started at http://127.0.0.1:46625/ using document root <none> and password file <none>
I20260806 03:46:11.057389 19026 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:11.057454 19026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:11.057639 19026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:11.058784 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "6d626af49b954fe68a3aa12d8c746e13"
format_stamp: "Formatted at 2026-08-06 03:46:11 on dist-test-slave-10pc"
server_key: "de28c2b6d300daf1cebf857c61c7322c"
server_key_iv: "0ec0f30780ba56e3c03a74fa0ee772a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:11.059156 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "6d626af49b954fe68a3aa12d8c746e13"
format_stamp: "Formatted at 2026-08-06 03:46:11 on dist-test-slave-10pc"
server_key: "de28c2b6d300daf1cebf857c61c7322c"
server_key_iv: "0ec0f30780ba56e3c03a74fa0ee772a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:11.060595 19026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:11.061420 19041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:11.061619 19026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:11.061744 19026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "6d626af49b954fe68a3aa12d8c746e13"
format_stamp: "Formatted at 2026-08-06 03:46:11 on dist-test-slave-10pc"
server_key: "de28c2b6d300daf1cebf857c61c7322c"
server_key_iv: "0ec0f30780ba56e3c03a74fa0ee772a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:11.061826 19026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:11.073204 19026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:11.075215 19026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:11.075390 19026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:11.079681 19026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43001
I20260806 03:46:11.079716 19094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43001 every 8 connection(s)
I20260806 03:46:11.080128 19026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:11.081099 19095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:11.083482 19095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13: Bootstrap starting.
I20260806 03:46:11.084046 19095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:11.084295 19095 log.cc:824] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:11.085055 19095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13: No bootstrap required, opened a new log
I20260806 03:46:11.086599 19095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626af49b954fe68a3aa12d8c746e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260806 03:46:11.086753 19095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:11.086777 19095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d626af49b954fe68a3aa12d8c746e13, State: Initialized, Role: FOLLOWER
I20260806 03:46:11.086885 19095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626af49b954fe68a3aa12d8c746e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260806 03:46:11.086941 19095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:11.086961 19095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:11.087034 19095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:11.087574 19095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626af49b954fe68a3aa12d8c746e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260806 03:46:11.087677 19095 leader_election.cc:302] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d626af49b954fe68a3aa12d8c746e13; no voters:
I20260806 03:46:11.087878 19095 leader_election.cc:288] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:11.087958 19098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:11.088120 19095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:11.088183 19098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [term 1 LEADER]: Becoming Leader. State: Replica: 6d626af49b954fe68a3aa12d8c746e13, State: Running, Role: LEADER
I20260806 03:46:11.088295 19098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626af49b954fe68a3aa12d8c746e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260806 03:46:11.088956 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d626af49b954fe68a3aa12d8c746e13. Latest consensus state: current_term: 1 leader_uuid: "6d626af49b954fe68a3aa12d8c746e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626af49b954fe68a3aa12d8c746e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } } }
I20260806 03:46:11.089068 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:11.089531 19107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:11.089701 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19026
I20260806 03:46:11.089823 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:11.089543 19099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d626af49b954fe68a3aa12d8c746e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626af49b954fe68a3aa12d8c746e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } } }
I20260806 03:46:11.089972 19099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:11.090042 30816 external_mini_cluster.cc:1428] Setting key f402e89cf92af0dbe495af564bed1806
I20260806 03:46:11.090094 19107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:11.091930 19107 catalog_manager.cc:1347] Generated new cluster ID: 5adb102ed9ff4a97824aa01440f57663
I20260806 03:46:11.091986 19107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:46:11 dist-test-slave-10pc krb5kdc[19013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:46:11.102578 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:11.090576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39015 (local address 127.0.0.1:43001)
0806 03:46:11.090756 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:11.090766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:11.090974 (+ 208us) server_negotiation.cc:407] Connection header received
0806 03:46:11.091111 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:11.091119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:11.091178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:11.091499 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:11.092323 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:11.093765 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:11.094895 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:11.095518 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:11.095635 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:11.098657 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:11.098662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:11.098663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:11.098677 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:11.100773 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:11.101221 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:11.101224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:11.101224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:11.101337 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:11.101580 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:11.101581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:11.101581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:11.101917 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:11.102094 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:11.102216 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 03:46:11.102323 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260806 03:46:11.103641 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:11.112641 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:11.105196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56231 (local address 127.0.0.1:43001)
0806 03:46:11.105234 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:11.105236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:11.105332 (+ 96us) server_negotiation.cc:407] Connection header received
0806 03:46:11.105379 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:11.105380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:11.105390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:11.105452 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:11.106039 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:11.106828 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:11.107818 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:11.108148 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:11.108231 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:11.109665 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:11.109667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:11.109667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:11.109675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:11.111311 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:11.111731 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:11.111732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:11.111732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:11.111795 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:11.112027 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:11.112028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:11.112029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:11.112210 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:11.112327 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:11.112470 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 03:46:11.112540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:46:11.116861 19107 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:11.117611 19107 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:11.121675 19107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d626af49b954fe68a3aa12d8c746e13: Generated new TSK 0
I20260806 03:46:11.122079 19107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:11.125293 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19026
W20260806 03:46:11.132304 19011 connection.cc:535] client connection to 127.0.0.1:43001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43001): Connection reset by peer (error 104)
2026-08-06T03:46:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:11 dist-test-slave-10pc krb5kdc[19127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:11 dist-test-slave-10pc krb5kdc[19127](info): set up 2 sockets
Aug 06 03:46:11 dist-test-slave-10pc krb5kdc[19127](info): commencing operation
krb5kdc: starting...
W20260806 03:46:12.157836 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:12 dist-test-slave-10pc krb5kdc[19127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:12.291759 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:12.376813 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:12.377041 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:12.377074 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:12.378187 19140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:12.378225 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:12.378239 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:12.378252 19140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:12.378265 19140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:12.379726 19140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:12.379940 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:12.380131 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:12.385134 19146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:12.385202 19148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:12.385182 19145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:12.385211 19140 server_base.cc:1034] running on GCE node
I20260806 03:46:12.385644 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:12.385943 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:12.387094 19140 hybrid_clock.cc:648] HybridClock initialized: now 1785987972387076 us; error 49 us; skew 500 ppm
Aug 06 03:46:12 dist-test-slave-10pc krb5kdc[19127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:12.390036 19140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:12.390627 19140 webserver.cc:460] Webserver started at http://127.0.0.1:38821/ using document root <none> and password file <none>
I20260806 03:46:12.390918 19140 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:12.391000 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:12.391242 19140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:12.392390 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "452fc561a5654326a8e1416513c87f22"
format_stamp: "Formatted at 2026-08-06 03:46:12 on dist-test-slave-10pc"
server_key: "b809fb86df136fb94888a72fd1541a28"
server_key_iv: "be662e1baa9425056d687082af613b97"
server_key_version: "encryptionkey@0"
I20260806 03:46:12.392831 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "452fc561a5654326a8e1416513c87f22"
format_stamp: "Formatted at 2026-08-06 03:46:12 on dist-test-slave-10pc"
server_key: "b809fb86df136fb94888a72fd1541a28"
server_key_iv: "be662e1baa9425056d687082af613b97"
server_key_version: "encryptionkey@0"
I20260806 03:46:12.394277 19140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:12.395102 19155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:12.395293 19140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:12.395401 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "452fc561a5654326a8e1416513c87f22"
format_stamp: "Formatted at 2026-08-06 03:46:12 on dist-test-slave-10pc"
server_key: "b809fb86df136fb94888a72fd1541a28"
server_key_iv: "be662e1baa9425056d687082af613b97"
server_key_version: "encryptionkey@0"
I20260806 03:46:12.395485 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:12.420373 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:12.422508 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:12.422683 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:12.426926 19140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33821
I20260806 03:46:12.426966 19207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33821 every 8 connection(s)
I20260806 03:46:12.427379 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:12.427843 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19140
I20260806 03:46:12.427989 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:12.428292 30816 external_mini_cluster.cc:1428] Setting key 9223d1acf539459362a28d05fb7e3002
I20260806 03:46:12.428551 19208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:12.431995 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22: Bootstrap starting.
I20260806 03:46:12.432780 19208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:12.433111 19208 log.cc:824] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:12.434020 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22: No bootstrap required, opened a new log
I20260806 03:46:12.434247 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:12.428911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59519 (local address 127.0.0.1:33821)
0806 03:46:12.429104 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:12.429112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:12.429137 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:46:12.429658 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:12.429665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:12.429712 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:12.429828 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:12.430086 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:12.430983 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:12.432060 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:12.432579 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:12.432663 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:12.433452 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:12.433457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:12.433457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:12.433679 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:12.433723 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:12.433735 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:12.433808 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:12.433932 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 03:46:12.433978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260806 03:46:12.435309 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:12.436817 19208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452fc561a5654326a8e1416513c87f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260806 03:46:12.437023 19208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:12.437089 19208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452fc561a5654326a8e1416513c87f22, State: Initialized, Role: FOLLOWER
I20260806 03:46:12.437232 19208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452fc561a5654326a8e1416513c87f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260806 03:46:12.437325 19208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:12.437361 19208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:12.437417 19208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:12.438071 19208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452fc561a5654326a8e1416513c87f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260806 03:46:12.438186 19208 leader_election.cc:302] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452fc561a5654326a8e1416513c87f22; no voters:
I20260806 03:46:12.438347 19208 leader_election.cc:288] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:12.438396 19218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:12.438592 19208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:12.438556 19218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [term 1 LEADER]: Becoming Leader. State: Replica: 452fc561a5654326a8e1416513c87f22, State: Running, Role: LEADER
I20260806 03:46:12.438720 19218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452fc561a5654326a8e1416513c87f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260806 03:46:12.439033 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452fc561a5654326a8e1416513c87f22. Latest consensus state: current_term: 1 leader_uuid: "452fc561a5654326a8e1416513c87f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452fc561a5654326a8e1416513c87f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } } }
I20260806 03:46:12.439096 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:12.439519 19219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452fc561a5654326a8e1416513c87f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452fc561a5654326a8e1416513c87f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } } }
I20260806 03:46:12.439607 19219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:12.440546 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:12.437000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54533 (local address 127.0.0.1:33821)
0806 03:46:12.437042 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:12.437044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:12.437170 (+ 126us) server_negotiation.cc:407] Connection header received
0806 03:46:12.437240 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:12.437241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:12.437251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:12.437301 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:12.437461 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:12.437940 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:12.438973 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:12.439297 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:12.439381 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:12.440112 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:12.440114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:12.440114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:12.440118 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:12.440136 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:12.440140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:12.440198 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:12.440434 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 03:46:12.440444 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260806 03:46:12.440580 19234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:12.440892 19234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:12.440954 19224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:12.441128 19224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:12.442641 19224 catalog_manager.cc:1347] Generated new cluster ID: 7204c6c8ac3c4f058bc616663095bf1e
I20260806 03:46:12.442711 19224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:12.452952 19224 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:12.453704 19224 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:12.465163 19224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452fc561a5654326a8e1416513c87f22: Generated new TSK 0
I20260806 03:46:12.465409 19224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:12.468505 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19140
W20260806 03:46:12.474990 19125 connection.cc:535] client connection to 127.0.0.1:33821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33821): Connection reset by peer (error 104)
2026-08-06T03:46:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:12 dist-test-slave-10pc krb5kdc[19239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:12 dist-test-slave-10pc krb5kdc[19239](info): set up 2 sockets
Aug 06 03:46:12 dist-test-slave-10pc krb5kdc[19239](info): commencing operation
krb5kdc: starting...
W20260806 03:46:13.521833 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:13 dist-test-slave-10pc krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:13.647895 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:13.732769 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:13.733023 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:13.733062 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:13.734233 19252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:13.734305 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:13.734333 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:13.734362 19252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:13.734387 19252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:13.735916 19252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:13.736217 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:13.736517 19252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:13.741922 19258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:13.741886 19260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:13.742023 19252 server_base.cc:1034] running on GCE node
W20260806 03:46:13.742187 19257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:13.742470 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:13.742782 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:13.743930 19252 hybrid_clock.cc:648] HybridClock initialized: now 1785987973743869 us; error 72 us; skew 500 ppm
Aug 06 03:46:13 dist-test-slave-10pc krb5kdc[19239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:13.746899 19252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:13.747550 19252 webserver.cc:460] Webserver started at http://127.0.0.1:42919/ using document root <none> and password file <none>
I20260806 03:46:13.747843 19252 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:13.747923 19252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:13.748106 19252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:13.749416 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "c123644592a9448d8172491d75d2fd2c"
format_stamp: "Formatted at 2026-08-06 03:46:13 on dist-test-slave-10pc"
server_key: "cb247e849f84f2d9c4c9e707657a874a"
server_key_iv: "b8d6dfa1a63493190fbc925b60299ca4"
server_key_version: "encryptionkey@0"
I20260806 03:46:13.749830 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "c123644592a9448d8172491d75d2fd2c"
format_stamp: "Formatted at 2026-08-06 03:46:13 on dist-test-slave-10pc"
server_key: "cb247e849f84f2d9c4c9e707657a874a"
server_key_iv: "b8d6dfa1a63493190fbc925b60299ca4"
server_key_version: "encryptionkey@0"
I20260806 03:46:13.751469 19252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:13.752310 19267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:13.752544 19252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:13.752667 19252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "c123644592a9448d8172491d75d2fd2c"
format_stamp: "Formatted at 2026-08-06 03:46:13 on dist-test-slave-10pc"
server_key: "cb247e849f84f2d9c4c9e707657a874a"
server_key_iv: "b8d6dfa1a63493190fbc925b60299ca4"
server_key_version: "encryptionkey@0"
I20260806 03:46:13.752759 19252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:13.756673 19252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:13.758745 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:13.758925 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:13.763692 19252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35469
I20260806 03:46:13.763769 19320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35469 every 8 connection(s)
I20260806 03:46:13.764169 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:13.764935 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:13.767237 19321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c: Bootstrap starting.
I20260806 03:46:13.767776 19321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:13.768010 19321 log.cc:824] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:13.768749 19321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c: No bootstrap required, opened a new log
I20260806 03:46:13.770301 19321 raft_consensus.cc:348] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123644592a9448d8172491d75d2fd2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260806 03:46:13.770440 19321 raft_consensus.cc:374] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:13.770462 19321 raft_consensus.cc:729] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c123644592a9448d8172491d75d2fd2c, State: Initialized, Role: FOLLOWER
I20260806 03:46:13.770601 19321 consensus_queue.cc:260] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123644592a9448d8172491d75d2fd2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260806 03:46:13.770684 19321 raft_consensus.cc:388] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:13.770707 19321 raft_consensus.cc:482] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:13.770736 19321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:13.771237 19321 raft_consensus.cc:504] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123644592a9448d8172491d75d2fd2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260806 03:46:13.771324 19321 leader_election.cc:302] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c123644592a9448d8172491d75d2fd2c; no voters:
I20260806 03:46:13.771456 19321 leader_election.cc:288] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:13.771541 19324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:13.771725 19324 raft_consensus.cc:686] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [term 1 LEADER]: Becoming Leader. State: Replica: c123644592a9448d8172491d75d2fd2c, State: Running, Role: LEADER
I20260806 03:46:13.771735 19321 sys_catalog.cc:564] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:13.771828 19324 consensus_queue.cc:237] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123644592a9448d8172491d75d2fd2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260806 03:46:13.772212 19326 sys_catalog.cc:455] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c123644592a9448d8172491d75d2fd2c. Latest consensus state: current_term: 1 leader_uuid: "c123644592a9448d8172491d75d2fd2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123644592a9448d8172491d75d2fd2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } } }
I20260806 03:46:13.772305 19326 sys_catalog.cc:458] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:13.772394 19325 sys_catalog.cc:455] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c123644592a9448d8172491d75d2fd2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123644592a9448d8172491d75d2fd2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } } }
I20260806 03:46:13.772511 19325 sys_catalog.cc:458] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:13.772639 19332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:13.773453 19332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:13.773871 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19252
I20260806 03:46:13.774019 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:13.774216 30816 external_mini_cluster.cc:1428] Setting key e10e54aeb5aed8f3eee3cd2d4f50ad60
I20260806 03:46:13.775063 19332 catalog_manager.cc:1347] Generated new cluster ID: 150e525ce5c442d4a343be71307f7b3e
I20260806 03:46:13.775115 19332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:13.776576 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:13.774718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42285 (local address 127.0.0.1:35469)
0806 03:46:13.774883 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:13.774892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:13.775026 (+ 134us) server_negotiation.cc:407] Connection header received
0806 03:46:13.775176 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:13.775184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:13.775219 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:13.775365 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:13.776118 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:13.776121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:13.776122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:13.776126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:13.776176 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:13.776191 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:13.776252 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:13.776350 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 03:46:13.776388 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260806 03:46:13.780988 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:13.783749 19332 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:13.783964 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:13.782682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43489 (local address 127.0.0.1:35469)
0806 03:46:13.782719 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:13.782721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:13.782864 (+ 143us) server_negotiation.cc:407] Connection header received
0806 03:46:13.782932 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:13.782933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:13.782951 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:13.783009 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:13.783722 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:13.783724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:13.783724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:13.783729 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:13.783745 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:13.783749 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:13.783800 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:13.783855 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 03:46:13.783874 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:13.783968 19332 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:13.795305 19332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c123644592a9448d8172491d75d2fd2c: Generated new TSK 0
I20260806 03:46:13.795552 19332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:13.805264 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19252
2026-08-06T03:46:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:13 dist-test-slave-10pc krb5kdc[19353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:13 dist-test-slave-10pc krb5kdc[19353](info): set up 2 sockets
Aug 06 03:46:13 dist-test-slave-10pc krb5kdc[19353](info): commencing operation
krb5kdc: starting...
W20260806 03:46:14.845642 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:14 dist-test-slave-10pc krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:14.972312 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:15.056691 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:15.056939 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:15.056979 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:15.058151 19366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:15.058231 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:15.058257 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:15.058279 19366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:15.058308 19366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:15.059820 19366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:15.060118 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:15.060361 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:15.065364 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:15.065335 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:15.065325 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:15.065523 19366 server_base.cc:1034] running on GCE node
I20260806 03:46:15.065806 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:15.066103 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:15.067248 19366 hybrid_clock.cc:648] HybridClock initialized: now 1785987975067226 us; error 36 us; skew 500 ppm
Aug 06 03:46:15 dist-test-slave-10pc krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:15.069801 19366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:15.070302 19366 webserver.cc:460] Webserver started at http://127.0.0.1:38667/ using document root <none> and password file <none>
I20260806 03:46:15.070547 19366 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:15.070616 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:15.070766 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:15.071857 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "d6a80d56dc4f4287b99b859ad4a9c849"
format_stamp: "Formatted at 2026-08-06 03:46:15 on dist-test-slave-10pc"
server_key: "1ce89c0ff61abd245fc9ab4538ef9cf9"
server_key_iv: "742db53d5f987bd8f6e800ad7306b080"
server_key_version: "encryptionkey@0"
I20260806 03:46:15.072270 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "d6a80d56dc4f4287b99b859ad4a9c849"
format_stamp: "Formatted at 2026-08-06 03:46:15 on dist-test-slave-10pc"
server_key: "1ce89c0ff61abd245fc9ab4538ef9cf9"
server_key_iv: "742db53d5f987bd8f6e800ad7306b080"
server_key_version: "encryptionkey@0"
I20260806 03:46:15.073787 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:15.074679 19381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:15.074899 19366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:15.075012 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "d6a80d56dc4f4287b99b859ad4a9c849"
format_stamp: "Formatted at 2026-08-06 03:46:15 on dist-test-slave-10pc"
server_key: "1ce89c0ff61abd245fc9ab4538ef9cf9"
server_key_iv: "742db53d5f987bd8f6e800ad7306b080"
server_key_version: "encryptionkey@0"
I20260806 03:46:15.075093 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:15.098543 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:15.100538 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:15.100710 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:15.105070 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40775
I20260806 03:46:15.105135 19434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40775 every 8 connection(s)
I20260806 03:46:15.105523 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:15.106405 19435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:15.108422 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19366
I20260806 03:46:15.108582 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:15.108799 30816 external_mini_cluster.cc:1428] Setting key 36c2b625dc30970e75e3816f12c5b6d3
I20260806 03:46:15.109014 19435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849: Bootstrap starting.
I20260806 03:46:15.109740 19435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:15.110050 19435 log.cc:824] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:15.110801 19435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849: No bootstrap required, opened a new log
I20260806 03:46:15.113256 19435 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260806 03:46:15.113436 19435 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:15.113468 19435 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a80d56dc4f4287b99b859ad4a9c849, State: Initialized, Role: FOLLOWER
I20260806 03:46:15.113600 19435 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260806 03:46:15.113677 19435 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:15.113708 19435 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:15.113752 19435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:15.114518 19435 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260806 03:46:15.114718 19435 leader_election.cc:302] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a80d56dc4f4287b99b859ad4a9c849; no voters:
I20260806 03:46:15.114936 19435 leader_election.cc:288] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:15.114959 19440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:15.115089 19440 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [term 1 LEADER]: Becoming Leader. State: Replica: d6a80d56dc4f4287b99b859ad4a9c849, State: Running, Role: LEADER
I20260806 03:46:15.115216 19435 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:15.115233 19440 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260806 03:46:15.115618 19441 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } } }
I20260806 03:46:15.115713 19441 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:15.115604 19440 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a80d56dc4f4287b99b859ad4a9c849. Latest consensus state: current_term: 1 leader_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a80d56dc4f4287b99b859ad4a9c849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } } }
I20260806 03:46:15.116039 19440 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849 [sys.catalog]: This master's current role is: LEADER
Aug 06 03:46:15 dist-test-slave-10pc krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:46:15.116734 19446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:15.117498 19446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:15.119442 19446 catalog_manager.cc:1347] Generated new cluster ID: 62102dcc735c4f9d8445d622d1c10ba0
I20260806 03:46:15.119491 19446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:15.120769 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:15.109308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57031 (local address 127.0.0.1:40775)
0806 03:46:15.109515 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:15.109527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:15.109564 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:46:15.110052 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:15.110060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:15.110109 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:15.110304 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:15.110979 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:15.112379 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:15.113254 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:15.113856 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:15.113991 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:15.116891 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:15.116896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:15.116896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:15.116908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:15.118881 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:15.119404 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:15.119408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:15.119408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:15.119523 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:15.119762 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:15.119763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:15.119763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:15.120087 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:15.120254 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:15.120363 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 03:46:15.120538 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260806 03:46:15.121974 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:15.127959 19446 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:15.128659 19446 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:15.131302 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:15.123482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46351 (local address 127.0.0.1:40775)
0806 03:46:15.123517 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:15.123518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:15.123586 (+ 68us) server_negotiation.cc:407] Connection header received
0806 03:46:15.123639 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:15.123640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:15.123649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:15.123724 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:15.124314 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:15.125480 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:15.126483 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:15.126807 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:15.126882 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:15.128339 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:15.128341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:15.128341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:15.128348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:15.130127 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:15.130508 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:15.130509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:15.130509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:15.130572 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:15.130773 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:15.130774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:15.130774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:15.130902 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:15.131001 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:15.131115 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 03:46:15.131209 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:46:15.134255 19446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a80d56dc4f4287b99b859ad4a9c849: Generated new TSK 0
I20260806 03:46:15.134425 19446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:15.138183 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19366
2026-08-06T03:46:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:15 dist-test-slave-10pc krb5kdc[19466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:15 dist-test-slave-10pc krb5kdc[19466](info): set up 2 sockets
Aug 06 03:46:15 dist-test-slave-10pc krb5kdc[19466](info): commencing operation
krb5kdc: starting...
W20260806 03:46:16.217833 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:16 dist-test-slave-10pc krb5kdc[19466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:16.348282 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:16.440003 19479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:16.440256 19479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:16.440282 19479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:16.441591 19479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:16.441642 19479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:16.441658 19479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:16.441672 19479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:16.441686 19479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:16.443296 19479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:16.443583 19479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:16.443799 19479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:16.449174 19484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:16.449184 19487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:16.449184 19485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:16.449249 19479 server_base.cc:1034] running on GCE node
I20260806 03:46:16.449662 19479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:16.449982 19479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:16.451203 19479 hybrid_clock.cc:648] HybridClock initialized: now 1785987976451163 us; error 42 us; skew 500 ppm
Aug 06 03:46:16 dist-test-slave-10pc krb5kdc[19466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:16.458781 19479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:16.459231 19479 webserver.cc:460] Webserver started at http://127.0.0.1:45831/ using document root <none> and password file <none>
I20260806 03:46:16.459435 19479 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:16.459477 19479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:16.459576 19479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:16.460794 19479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "46de4a68997a474c9aa639e4508daa58"
format_stamp: "Formatted at 2026-08-06 03:46:16 on dist-test-slave-10pc"
server_key: "3396aefe961fc010cd53e6b427f0fe8e"
server_key_iv: "b14a1e0bd8d03201a1d7bdf90e1d66e5"
server_key_version: "encryptionkey@0"
I20260806 03:46:16.461134 19479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "46de4a68997a474c9aa639e4508daa58"
format_stamp: "Formatted at 2026-08-06 03:46:16 on dist-test-slave-10pc"
server_key: "3396aefe961fc010cd53e6b427f0fe8e"
server_key_iv: "b14a1e0bd8d03201a1d7bdf90e1d66e5"
server_key_version: "encryptionkey@0"
I20260806 03:46:16.462515 19479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:16.463291 19494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:16.463490 19479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:16.463578 19479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "46de4a68997a474c9aa639e4508daa58"
format_stamp: "Formatted at 2026-08-06 03:46:16 on dist-test-slave-10pc"
server_key: "3396aefe961fc010cd53e6b427f0fe8e"
server_key_iv: "b14a1e0bd8d03201a1d7bdf90e1d66e5"
server_key_version: "encryptionkey@0"
I20260806 03:46:16.463639 19479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:16.473177 19479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:16.475538 19479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:16.475653 19479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:16.479684 19479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44299
I20260806 03:46:16.479751 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44299 every 8 connection(s)
I20260806 03:46:16.480126 19479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:16.481081 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:16.483482 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58: Bootstrap starting.
I20260806 03:46:16.484108 19548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:16.484174 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19479
I20260806 03:46:16.484319 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:16.484556 30816 external_mini_cluster.cc:1428] Setting key 19bc84d4bc35ea3ae779cc9e0ddad4a4
I20260806 03:46:16.484561 19548 log.cc:824] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:16.485419 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58: No bootstrap required, opened a new log
I20260806 03:46:16.487601 19548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46de4a68997a474c9aa639e4508daa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260806 03:46:16.487768 19548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:16.487833 19548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46de4a68997a474c9aa639e4508daa58, State: Initialized, Role: FOLLOWER
I20260806 03:46:16.488009 19548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46de4a68997a474c9aa639e4508daa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260806 03:46:16.488109 19548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:16.488184 19548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:16.488241 19548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:16.488891 19548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46de4a68997a474c9aa639e4508daa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260806 03:46:16.489017 19548 leader_election.cc:302] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46de4a68997a474c9aa639e4508daa58; no voters:
I20260806 03:46:16.489216 19548 leader_election.cc:288] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:16.489259 19553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:16.489406 19553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [term 1 LEADER]: Becoming Leader. State: Replica: 46de4a68997a474c9aa639e4508daa58, State: Running, Role: LEADER
I20260806 03:46:16.489478 19548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:16.489500 19553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46de4a68997a474c9aa639e4508daa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260806 03:46:16.489892 19554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46de4a68997a474c9aa639e4508daa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46de4a68997a474c9aa639e4508daa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } } }
I20260806 03:46:16.489991 19554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:16.490253 19559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:16.490556 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:16.485157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60971 (local address 127.0.0.1:44299)
0806 03:46:16.485340 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:16.485351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:16.485388 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:46:16.485889 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:16.485896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:16.485946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:16.486097 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:16.486295 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:16.487213 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:16.488136 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:16.488714 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:16.488839 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:16.489588 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:16.489592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:16.489593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:16.489822 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:16.489861 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:16.489875 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:16.489949 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:16.490286 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 03:46:16.490360 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260806 03:46:16.489898 19555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46de4a68997a474c9aa639e4508daa58. Latest consensus state: current_term: 1 leader_uuid: "46de4a68997a474c9aa639e4508daa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46de4a68997a474c9aa639e4508daa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } } }
I20260806 03:46:16.490839 19555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:16.491250 19559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:16.491904 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:16.496572 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:16.493564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55023 (local address 127.0.0.1:44299)
0806 03:46:16.493608 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:16.493609 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:16.493660 (+ 51us) server_negotiation.cc:407] Connection header received
0806 03:46:16.493734 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:16.493734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:16.493743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:16.493789 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:16.493923 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:16.494384 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:16.495399 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:16.495580 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:16.495629 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:16.496269 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:16.496270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:16.496270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:16.496274 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:16.496291 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:16.496295 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:16.496354 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:16.496430 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 03:46:16.496490 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:46:16.497586 19559 catalog_manager.cc:1347] Generated new cluster ID: b3f9d1ac2c3f4efb9d227a6f7ae3627a
I20260806 03:46:16.497629 19559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:16.513114 19559 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:16.513690 19559 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:16.522140 19559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46de4a68997a474c9aa639e4508daa58: Generated new TSK 0
I20260806 03:46:16.522364 19559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:16.530339 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19479
2026-08-06T03:46:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:16 dist-test-slave-10pc krb5kdc[19579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:16 dist-test-slave-10pc krb5kdc[19579](info): set up 2 sockets
Aug 06 03:46:16 dist-test-slave-10pc krb5kdc[19579](info): commencing operation
krb5kdc: starting...
W20260806 03:46:17.577692 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:17 dist-test-slave-10pc krb5kdc[19579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:17.707424 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:17.792176 19592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:17.792389 19592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:17.792415 19592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:17.793639 19592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:17.793689 19592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:17.793704 19592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:17.793717 19592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:17.793730 19592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:17.795244 19592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:17.795522 19592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:17.795761 19592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:17.800813 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:17.801030 19592 server_base.cc:1034] running on GCE node
W20260806 03:46:17.801138 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:17.801187 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:17.801630 19592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:17.801959 19592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:17.803161 19592 hybrid_clock.cc:648] HybridClock initialized: now 1785987977803163 us; error 60 us; skew 500 ppm
Aug 06 03:46:17 dist-test-slave-10pc krb5kdc[19579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:17.805807 19592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:17.806295 19592 webserver.cc:460] Webserver started at http://127.0.0.1:42857/ using document root <none> and password file <none>
I20260806 03:46:17.806537 19592 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:17.806612 19592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:17.806763 19592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:17.807988 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "9f33742dd5d14ecda2680fa333ee4c07"
format_stamp: "Formatted at 2026-08-06 03:46:17 on dist-test-slave-10pc"
server_key: "fd724dcf4464a1cc4d848078f423e124"
server_key_iv: "d5e060af599cbdbc3fec5649354423b7"
server_key_version: "encryptionkey@0"
I20260806 03:46:17.808379 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "9f33742dd5d14ecda2680fa333ee4c07"
format_stamp: "Formatted at 2026-08-06 03:46:17 on dist-test-slave-10pc"
server_key: "fd724dcf4464a1cc4d848078f423e124"
server_key_iv: "d5e060af599cbdbc3fec5649354423b7"
server_key_version: "encryptionkey@0"
I20260806 03:46:17.809783 19592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:17.810554 19607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:17.810725 19592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 03:46:17.810832 19592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "9f33742dd5d14ecda2680fa333ee4c07"
format_stamp: "Formatted at 2026-08-06 03:46:17 on dist-test-slave-10pc"
server_key: "fd724dcf4464a1cc4d848078f423e124"
server_key_iv: "d5e060af599cbdbc3fec5649354423b7"
server_key_version: "encryptionkey@0"
I20260806 03:46:17.810909 19592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:17.814334 19592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:17.816670 19592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:17.816828 19592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:17.820888 19592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39109
I20260806 03:46:17.820955 19660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39109 every 8 connection(s)
I20260806 03:46:17.821302 19592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:17.822397 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:17.822801 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19592
I20260806 03:46:17.822950 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:17.823161 30816 external_mini_cluster.cc:1428] Setting key d75867e56e4e8be667aeaa52de09cb0e
I20260806 03:46:17.825558 19661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07: Bootstrap starting.
I20260806 03:46:17.825754 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:17.823742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59259 (local address 127.0.0.1:39109)
0806 03:46:17.823918 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:17.823927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:17.823952 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:46:17.824377 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:17.824383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:17.824407 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:17.824593 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:17.825292 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:17.825296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:17.825296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:17.825300 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:17.825336 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:17.825346 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:17.825403 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:17.825498 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 03:46:17.825558 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260806 03:46:17.826295 19661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:17.826597 19661 log.cc:824] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:17.826722 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:17.827447 19661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07: No bootstrap required, opened a new log
I20260806 03:46:17.829265 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:17.828259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52161 (local address 127.0.0.1:39109)
0806 03:46:17.828295 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:17.828297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:17.828346 (+ 49us) server_negotiation.cc:407] Connection header received
0806 03:46:17.828424 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:17.828425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:17.828429 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:17.828539 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:17.829081 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:17.829082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:17.829083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:17.829086 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:17.829099 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:17.829102 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:17.829142 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:17.829199 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 03:46:17.829211 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 03:46:17.830307 19661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f33742dd5d14ecda2680fa333ee4c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260806 03:46:17.830453 19661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:17.830507 19661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f33742dd5d14ecda2680fa333ee4c07, State: Initialized, Role: FOLLOWER
I20260806 03:46:17.830636 19661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f33742dd5d14ecda2680fa333ee4c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260806 03:46:17.830724 19661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:17.830747 19661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:17.830799 19661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:17.831386 19661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f33742dd5d14ecda2680fa333ee4c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260806 03:46:17.831507 19661 leader_election.cc:302] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f33742dd5d14ecda2680fa333ee4c07; no voters:
I20260806 03:46:17.831676 19661 leader_election.cc:288] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:17.831763 19671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:17.831939 19661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:17.831923 19671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [term 1 LEADER]: Becoming Leader. State: Replica: 9f33742dd5d14ecda2680fa333ee4c07, State: Running, Role: LEADER
I20260806 03:46:17.832082 19671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f33742dd5d14ecda2680fa333ee4c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260806 03:46:17.832522 19673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f33742dd5d14ecda2680fa333ee4c07. Latest consensus state: current_term: 1 leader_uuid: "9f33742dd5d14ecda2680fa333ee4c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f33742dd5d14ecda2680fa333ee4c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } } }
I20260806 03:46:17.832446 19672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f33742dd5d14ecda2680fa333ee4c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f33742dd5d14ecda2680fa333ee4c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } } }
I20260806 03:46:17.832640 19673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:17.832651 19672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:17.832911 19680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:17.833469 19680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:17.835273 19680 catalog_manager.cc:1347] Generated new cluster ID: e95eafa45f734b219e2011ef4ccf6432
I20260806 03:46:17.835331 19680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:17.842090 19680 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:17.842286 19680 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:17.845441 19680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f33742dd5d14ecda2680fa333ee4c07: Generated new TSK 0
I20260806 03:46:17.845623 19680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:17.851083 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19592
2026-08-06T03:46:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:17 dist-test-slave-10pc krb5kdc[19692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:17 dist-test-slave-10pc krb5kdc[19692](info): set up 2 sockets
Aug 06 03:46:17 dist-test-slave-10pc krb5kdc[19692](info): commencing operation
krb5kdc: starting...
W20260806 03:46:18.905743 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:18 dist-test-slave-10pc krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:19.035485 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:19.120573 19705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:19.120824 19705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:19.120863 19705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:19.122078 19705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:19.122156 19705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:19.122180 19705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:19.122200 19705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:19.122231 19705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:19.123777 19705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:19.124080 19705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:19.124336 19705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:19.129427 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:19.129433 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:19.129428 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:19.129745 19705 server_base.cc:1034] running on GCE node
I20260806 03:46:19.130024 19705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:19.130344 19705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:19.131513 19705 hybrid_clock.cc:648] HybridClock initialized: now 1785987979131475 us; error 53 us; skew 500 ppm
Aug 06 03:46:19 dist-test-slave-10pc krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:19.134341 19705 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:19.134912 19705 webserver.cc:460] Webserver started at http://127.0.0.1:34203/ using document root <none> and password file <none>
I20260806 03:46:19.135236 19705 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:19.135314 19705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:19.135468 19705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:19.136766 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "b0c481f2c8ff4335883751374fc3beda"
format_stamp: "Formatted at 2026-08-06 03:46:19 on dist-test-slave-10pc"
server_key: "f0e9ad8751fdc99b4b7863bfad66ec37"
server_key_iv: "0fde6c2dbb59bd29084f4cd41323b541"
server_key_version: "encryptionkey@0"
I20260806 03:46:19.137172 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "b0c481f2c8ff4335883751374fc3beda"
format_stamp: "Formatted at 2026-08-06 03:46:19 on dist-test-slave-10pc"
server_key: "f0e9ad8751fdc99b4b7863bfad66ec37"
server_key_iv: "0fde6c2dbb59bd29084f4cd41323b541"
server_key_version: "encryptionkey@0"
I20260806 03:46:19.138693 19705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:19.139559 19721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:19.139739 19705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:19.139863 19705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "b0c481f2c8ff4335883751374fc3beda"
format_stamp: "Formatted at 2026-08-06 03:46:19 on dist-test-slave-10pc"
server_key: "f0e9ad8751fdc99b4b7863bfad66ec37"
server_key_iv: "0fde6c2dbb59bd29084f4cd41323b541"
server_key_version: "encryptionkey@0"
I20260806 03:46:19.139947 19705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:19.168780 19705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:19.170887 19705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:19.171092 19705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:19.175688 19705 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39053
I20260806 03:46:19.175726 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39053 every 8 connection(s)
I20260806 03:46:19.176137 19705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:19.177124 19774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:19.179476 19774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda: Bootstrap starting.
I20260806 03:46:19.180078 19774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:19.180382 19774 log.cc:824] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:19.181176 19774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda: No bootstrap required, opened a new log
I20260806 03:46:19.181439 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19705
I20260806 03:46:19.181557 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:19.181764 30816 external_mini_cluster.cc:1428] Setting key dac387ad7bd7e3b161524995874cc61d
I20260806 03:46:19.183682 19774 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c481f2c8ff4335883751374fc3beda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260806 03:46:19.183877 19774 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:19.183917 19774 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c481f2c8ff4335883751374fc3beda, State: Initialized, Role: FOLLOWER
I20260806 03:46:19.184047 19774 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c481f2c8ff4335883751374fc3beda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260806 03:46:19.184166 19774 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:19.184216 19774 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:19.184267 19774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:19.184882 19774 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c481f2c8ff4335883751374fc3beda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260806 03:46:19.184974 19774 leader_election.cc:302] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c481f2c8ff4335883751374fc3beda; no voters:
I20260806 03:46:19.185184 19774 leader_election.cc:288] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:19.185227 19780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:19.185412 19780 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [term 1 LEADER]: Becoming Leader. State: Replica: b0c481f2c8ff4335883751374fc3beda, State: Running, Role: LEADER
I20260806 03:46:19.185462 19774 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:19.185520 19780 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c481f2c8ff4335883751374fc3beda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260806 03:46:19.185936 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c481f2c8ff4335883751374fc3beda. Latest consensus state: current_term: 1 leader_uuid: "b0c481f2c8ff4335883751374fc3beda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c481f2c8ff4335883751374fc3beda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } } }
I20260806 03:46:19.186070 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:19.186048 19781 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c481f2c8ff4335883751374fc3beda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c481f2c8ff4335883751374fc3beda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } } }
I20260806 03:46:19.186199 19781 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:19.186477 19789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:19.187235 19789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:19.188912 19789 catalog_manager.cc:1347] Generated new cluster ID: 3698074ec28142719aa3c670bf0bbd22
I20260806 03:46:19.188977 19789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 03:46:19 dist-test-slave-10pc krb5kdc[19692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:46:19.194963 19789 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:19.195165 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:19.182348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:42705 (local address 127.0.0.1:39053)
0806 03:46:19.182552 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:19.182560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:19.182587 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:46:19.183162 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:19.183170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:19.183221 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:19.183338 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:19.184130 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:19.185818 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:19.186952 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:19.187721 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:19.187831 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:19.190836 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:19.190841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:19.190842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:19.190855 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:19.192924 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:19.193410 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:19.193414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:19.193414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:19.193532 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:19.193797 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:19.193798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:19.193798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:19.194122 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:19.194324 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:19.194780 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 03:46:19.194889 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260806 03:46:19.195662 19789 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:19.196260 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:19.202684 19789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c481f2c8ff4335883751374fc3beda: Generated new TSK 0
I20260806 03:46:19.202943 19789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:19.205390 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:19.197766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:56239 (local address 127.0.0.1:39053)
0806 03:46:19.197803 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:19.197805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:19.197938 (+ 133us) server_negotiation.cc:407] Connection header received
0806 03:46:19.197971 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:19.197972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:19.197984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:19.198051 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:19.198643 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:19.199475 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:19.200515 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:19.200855 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:19.200936 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:19.202534 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:19.202535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:19.202536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:19.202543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:19.204194 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:19.204609 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:19.204610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:19.204611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:19.204675 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:19.204886 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:19.204887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:19.204887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:19.205029 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:19.205145 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:19.205234 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 03:46:19.205298 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:19.207491 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19705
W20260806 03:46:19.214105 19690 connection.cc:535] client connection to 127.0.0.1:39053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39053): Connection reset by peer (error 104)
2026-08-06T03:46:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:19 dist-test-slave-10pc krb5kdc[19807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:19 dist-test-slave-10pc krb5kdc[19807](info): set up 2 sockets
Aug 06 03:46:19 dist-test-slave-10pc krb5kdc[19807](info): commencing operation
krb5kdc: starting...
W20260806 03:46:20.277873 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:20 dist-test-slave-10pc krb5kdc[19807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:20.411728 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:20.496526 19820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:20.496764 19820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:20.496789 19820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:20.497901 19820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:20.497937 19820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:20.497952 19820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:20.497965 19820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:20.497977 19820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:20.499368 19820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:20.499568 19820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:20.499734 19820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:20.505119 19825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:20.505116 19828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:20.505182 19826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:20.505460 19820 server_base.cc:1034] running on GCE node
I20260806 03:46:20.505790 19820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:20.506040 19820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:20.507215 19820 hybrid_clock.cc:648] HybridClock initialized: now 1785987980507203 us; error 42 us; skew 500 ppm
Aug 06 03:46:20 dist-test-slave-10pc krb5kdc[19807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:20.509754 19820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:20.510239 19820 webserver.cc:460] Webserver started at http://127.0.0.1:33613/ using document root <none> and password file <none>
I20260806 03:46:20.510473 19820 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:20.510542 19820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:20.510703 19820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:20.511797 19820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "80d15b7310274ef19f935a615409dfa8"
format_stamp: "Formatted at 2026-08-06 03:46:20 on dist-test-slave-10pc"
server_key: "2f8ebbbd6b29ad52565ba9565a7d8ba0"
server_key_iv: "e15a76c6736017ba282142266563f753"
server_key_version: "encryptionkey@0"
I20260806 03:46:20.512176 19820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "80d15b7310274ef19f935a615409dfa8"
format_stamp: "Formatted at 2026-08-06 03:46:20 on dist-test-slave-10pc"
server_key: "2f8ebbbd6b29ad52565ba9565a7d8ba0"
server_key_iv: "e15a76c6736017ba282142266563f753"
server_key_version: "encryptionkey@0"
I20260806 03:46:20.513691 19820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:46:20.514488 19835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:20.514683 19820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:20.514787 19820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "80d15b7310274ef19f935a615409dfa8"
format_stamp: "Formatted at 2026-08-06 03:46:20 on dist-test-slave-10pc"
server_key: "2f8ebbbd6b29ad52565ba9565a7d8ba0"
server_key_iv: "e15a76c6736017ba282142266563f753"
server_key_version: "encryptionkey@0"
I20260806 03:46:20.514873 19820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:20.534603 19820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:20.536476 19820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:20.536654 19820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:20.540987 19820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35525
I20260806 03:46:20.541045 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35525 every 8 connection(s)
I20260806 03:46:20.541438 19820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:20.542320 19889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:20.544775 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8: Bootstrap starting.
I20260806 03:46:20.545393 19889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:20.545699 19889 log.cc:824] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:20.546433 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8: No bootstrap required, opened a new log
I20260806 03:46:20.547348 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19820
I20260806 03:46:20.547472 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:20.547683 30816 external_mini_cluster.cc:1428] Setting key 05a49197410387787c71837c7057a18a
I20260806 03:46:20.548789 19889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d15b7310274ef19f935a615409dfa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260806 03:46:20.548969 19889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:20.549010 19889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d15b7310274ef19f935a615409dfa8, State: Initialized, Role: FOLLOWER
I20260806 03:46:20.549206 19889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d15b7310274ef19f935a615409dfa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260806 03:46:20.549320 19889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:20.549366 19889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:20.549418 19889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:20.549976 19889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d15b7310274ef19f935a615409dfa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260806 03:46:20.550099 19889 leader_election.cc:302] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d15b7310274ef19f935a615409dfa8; no voters:
I20260806 03:46:20.550297 19889 leader_election.cc:288] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:20.550319 19894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:20.550436 19894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [term 1 LEADER]: Becoming Leader. State: Replica: 80d15b7310274ef19f935a615409dfa8, State: Running, Role: LEADER
I20260806 03:46:20.550573 19889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:20.550578 19894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d15b7310274ef19f935a615409dfa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260806 03:46:20.551105 19896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d15b7310274ef19f935a615409dfa8. Latest consensus state: current_term: 1 leader_uuid: "80d15b7310274ef19f935a615409dfa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d15b7310274ef19f935a615409dfa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260806 03:46:20.551216 19896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:20.551438 19895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80d15b7310274ef19f935a615409dfa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d15b7310274ef19f935a615409dfa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260806 03:46:20.551543 19895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:20.551813 19903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:20.552552 19903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:20.554114 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:20.548254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:47275 (local address 127.0.0.1:35525)
0806 03:46:20.548442 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:20.548474 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:20.548511 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:46:20.548773 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:20.548779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:20.548838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:20.548952 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:20.549231 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:20.550635 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:20.551765 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:20.552428 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:20.552563 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:20.553346 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:20.553351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:20.553351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:20.553541 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:20.553570 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:20.553581 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:20.553687 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:20.553870 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 03:46:20.553899 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260806 03:46:20.554279 19903 catalog_manager.cc:1347] Generated new cluster ID: de6f639c2dc241ffbcaf91434b4b7fb4
I20260806 03:46:20.554402 19903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:20.555328 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:20.560398 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:20.556886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:45935 (local address 127.0.0.1:35525)
0806 03:46:20.556920 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:20.556922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:20.556986 (+ 64us) server_negotiation.cc:407] Connection header received
0806 03:46:20.557053 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:20.557054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:20.557064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:20.557115 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:20.557245 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:20.557939 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:20.558939 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:20.559340 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:20.559424 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:20.560120 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:20.560121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:20.560121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:20.560126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:20.560149 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:20.560154 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:20.560218 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:20.560308 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 03:46:20.560321 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:46:20.563443 19903 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:20.563946 19903 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:20.568852 19903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80d15b7310274ef19f935a615409dfa8: Generated new TSK 0
I20260806 03:46:20.569036 19903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:20.574396 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19820
W20260806 03:46:20.580199 19805 connection.cc:535] client connection to 127.0.0.1:35525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35525): Connection reset by peer (error 104)
2026-08-06T03:46:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:20 dist-test-slave-10pc krb5kdc[19920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:20 dist-test-slave-10pc krb5kdc[19920](info): set up 2 sockets
Aug 06 03:46:20 dist-test-slave-10pc krb5kdc[19920](info): commencing operation
krb5kdc: starting...
W20260806 03:46:21.626211 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:21 dist-test-slave-10pc krb5kdc[19920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:21.757543 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:21.840965 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:21.841224 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:21.841262 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:21.842418 19933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:21.842499 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:21.842522 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:21.842543 19933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:21.842573 19933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:21.844116 19933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:21.844421 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:21.844689 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:21.849618 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:21.849622 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:21.849623 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:21.849960 19933 server_base.cc:1034] running on GCE node
I20260806 03:46:21.850258 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:21.850584 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:21.851768 19933 hybrid_clock.cc:648] HybridClock initialized: now 1785987981851768 us; error 47 us; skew 500 ppm
Aug 06 03:46:21 dist-test-slave-10pc krb5kdc[19920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:21.858999 19933 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:21.859509 19933 webserver.cc:460] Webserver started at http://127.0.0.1:34323/ using document root <none> and password file <none>
I20260806 03:46:21.859768 19933 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:21.859858 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:21.860013 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:21.861263 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "3af9cdf371b14965b6da28d35cc483e3"
format_stamp: "Formatted at 2026-08-06 03:46:21 on dist-test-slave-10pc"
server_key: "448f3259fe14ed44bd7c15f9ee28fc07"
server_key_iv: "70adb3019e314c169bae88dfdb68d450"
server_key_version: "encryptionkey@0"
I20260806 03:46:21.861651 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "3af9cdf371b14965b6da28d35cc483e3"
format_stamp: "Formatted at 2026-08-06 03:46:21 on dist-test-slave-10pc"
server_key: "448f3259fe14ed44bd7c15f9ee28fc07"
server_key_iv: "70adb3019e314c169bae88dfdb68d450"
server_key_version: "encryptionkey@0"
I20260806 03:46:21.863049 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 03:46:21.863781 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:21.863960 19933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:21.864074 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "3af9cdf371b14965b6da28d35cc483e3"
format_stamp: "Formatted at 2026-08-06 03:46:21 on dist-test-slave-10pc"
server_key: "448f3259fe14ed44bd7c15f9ee28fc07"
server_key_iv: "70adb3019e314c169bae88dfdb68d450"
server_key_version: "encryptionkey@0"
I20260806 03:46:21.864223 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:21.867893 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:21.870234 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:21.870344 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:21.874383 19933 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34543
I20260806 03:46:21.874420 20002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34543 every 8 connection(s)
I20260806 03:46:21.874775 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:21.875815 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:21.878185 20003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3: Bootstrap starting.
I20260806 03:46:21.878782 20003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:21.879065 20003 log.cc:824] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:21.879887 20003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3: No bootstrap required, opened a new log
I20260806 03:46:21.881563 20003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9cdf371b14965b6da28d35cc483e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20260806 03:46:21.881714 20003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:21.881738 20003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af9cdf371b14965b6da28d35cc483e3, State: Initialized, Role: FOLLOWER
I20260806 03:46:21.881908 20003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9cdf371b14965b6da28d35cc483e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20260806 03:46:21.881971 20003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:21.881992 20003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:21.882062 20003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:21.882598 20003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9cdf371b14965b6da28d35cc483e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20260806 03:46:21.882720 20003 leader_election.cc:302] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af9cdf371b14965b6da28d35cc483e3; no voters:
I20260806 03:46:21.882910 20003 leader_election.cc:288] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:21.882937 20006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:21.883054 20006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [term 1 LEADER]: Becoming Leader. State: Replica: 3af9cdf371b14965b6da28d35cc483e3, State: Running, Role: LEADER
I20260806 03:46:21.883037 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 19933
I20260806 03:46:21.883181 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:21.883190 20003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:21.883215 20006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9cdf371b14965b6da28d35cc483e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } }
I20260806 03:46:21.883385 30816 external_mini_cluster.cc:1428] Setting key 6ea51873d43ec76e97563fd3c402d62d
I20260806 03:46:21.883771 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af9cdf371b14965b6da28d35cc483e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9cdf371b14965b6da28d35cc483e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } } }
I20260806 03:46:21.883873 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:21.884220 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af9cdf371b14965b6da28d35cc483e3. Latest consensus state: current_term: 1 leader_uuid: "3af9cdf371b14965b6da28d35cc483e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9cdf371b14965b6da28d35cc483e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34543 } } }
I20260806 03:46:21.884312 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3 [sys.catalog]: This master's current role is: LEADER
W20260806 03:46:21.885205 20025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:21.885313 20025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:21.885381 20016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:21.885550 20016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:21.886292 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:21.884393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:51991 (local address 127.0.0.1:34543)
0806 03:46:21.884676 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:21.884687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:21.884725 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:46:21.884861 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:21.884867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:21.884892 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:21.885073 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:21.885865 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:21.885869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:21.885869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:21.885874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:21.885926 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:21.885938 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:21.886018 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:21.886087 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:46:21.886132 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":132,"threads_started":1}
I20260806 03:46:21.887113 20016 catalog_manager.cc:1347] Generated new cluster ID: 4d36f3c6a98d4c0295cdddb0b463a59c
I20260806 03:46:21.887169 20016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:21.887395 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:21.889961 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:21.888863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:44891 (local address 127.0.0.1:34543)
0806 03:46:21.888894 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:21.888896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:21.889018 (+ 122us) server_negotiation.cc:407] Connection header received
0806 03:46:21.889080 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:21.889081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:21.889086 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:21.889134 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:21.889743 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:21.889744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:21.889744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:21.889748 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:21.889764 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:21.889768 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:21.889821 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:21.889890 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:46:21.889903 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260806 03:46:21.901474 20016 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:21.901691 20016 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:21.906682 20016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af9cdf371b14965b6da28d35cc483e3: Generated new TSK 0
I20260806 03:46:21.906898 20016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:21.910260 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 19933
2026-08-06T03:46:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:21 dist-test-slave-10pc krb5kdc[20035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:21 dist-test-slave-10pc krb5kdc[20035](info): set up 2 sockets
Aug 06 03:46:21 dist-test-slave-10pc krb5kdc[20035](info): commencing operation
krb5kdc: starting...
W20260806 03:46:22.990291 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:23 dist-test-slave-10pc krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:23.117408 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:23.202538 20048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:23.202756 20048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:23.202780 20048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:23.204497 20048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:23.204566 20048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:23.204588 20048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:23.204607 20048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:23.204623 20048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:23.206231 20048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:23.206442 20048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:23.206632 20048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:23.211853 20053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:23.211859 20054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:23.211866 20048 server_base.cc:1034] running on GCE node
W20260806 03:46:23.211833 20056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:23.212400 20048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:23.212745 20048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:23.213899 20048 hybrid_clock.cc:648] HybridClock initialized: now 1785987983213884 us; error 29 us; skew 500 ppm
Aug 06 03:46:23 dist-test-slave-10pc krb5kdc[20035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:23.216338 20048 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:23.216871 20048 webserver.cc:460] Webserver started at http://127.0.0.1:36121/ using document root <none> and password file <none>
I20260806 03:46:23.217110 20048 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:23.217159 20048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:23.217315 20048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:23.218426 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "846079a6361f4f10af4995b3c4fff1b9"
format_stamp: "Formatted at 2026-08-06 03:46:23 on dist-test-slave-10pc"
server_key: "4b6dafce2f253a6331c9f3e0e907c0bb"
server_key_iv: "190ff4bf905b034db744413c121be6bf"
server_key_version: "encryptionkey@0"
I20260806 03:46:23.218812 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "846079a6361f4f10af4995b3c4fff1b9"
format_stamp: "Formatted at 2026-08-06 03:46:23 on dist-test-slave-10pc"
server_key: "4b6dafce2f253a6331c9f3e0e907c0bb"
server_key_iv: "190ff4bf905b034db744413c121be6bf"
server_key_version: "encryptionkey@0"
I20260806 03:46:23.220194 20048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:46:23.221057 20063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:23.221268 20048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:46:23.221365 20048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "846079a6361f4f10af4995b3c4fff1b9"
format_stamp: "Formatted at 2026-08-06 03:46:23 on dist-test-slave-10pc"
server_key: "4b6dafce2f253a6331c9f3e0e907c0bb"
server_key_iv: "190ff4bf905b034db744413c121be6bf"
server_key_version: "encryptionkey@0"
I20260806 03:46:23.221443 20048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:23.235483 20048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:23.237598 20048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:23.237774 20048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:23.241935 20048 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33983
I20260806 03:46:23.241992 20115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33983 every 8 connection(s)
I20260806 03:46:23.242381 20048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:23.242839 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20048
I20260806 03:46:23.242978 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:23.243227 30816 external_mini_cluster.cc:1428] Setting key 614785e4050f10491be3d9cac32dea91
I20260806 03:46:23.243533 20116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:23.246480 20116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9: Bootstrap starting.
I20260806 03:46:23.247298 20116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:23.247661 20116 log.cc:824] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:23.248499 20116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9: No bootstrap required, opened a new log
Aug 06 03:46:23 dist-test-slave-10pc krb5kdc[20035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:46:23.250734 20116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846079a6361f4f10af4995b3c4fff1b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260806 03:46:23.250877 20116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:23.250900 20116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846079a6361f4f10af4995b3c4fff1b9, State: Initialized, Role: FOLLOWER
I20260806 03:46:23.251049 20116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846079a6361f4f10af4995b3c4fff1b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260806 03:46:23.251142 20116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:23.251192 20116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:23.251250 20116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:23.252030 20116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846079a6361f4f10af4995b3c4fff1b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260806 03:46:23.252174 20116 leader_election.cc:302] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846079a6361f4f10af4995b3c4fff1b9; no voters:
I20260806 03:46:23.252364 20116 leader_election.cc:288] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:23.252429 20121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:23.252599 20121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [term 1 LEADER]: Becoming Leader. State: Replica: 846079a6361f4f10af4995b3c4fff1b9, State: Running, Role: LEADER
I20260806 03:46:23.252676 20116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:23.252779 20121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846079a6361f4f10af4995b3c4fff1b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260806 03:46:23.253677 20123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846079a6361f4f10af4995b3c4fff1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846079a6361f4f10af4995b3c4fff1b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } } }
I20260806 03:46:23.253775 20123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:23.254140 20133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:23.254290 20121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846079a6361f4f10af4995b3c4fff1b9. Latest consensus state: current_term: 1 leader_uuid: "846079a6361f4f10af4995b3c4fff1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846079a6361f4f10af4995b3c4fff1b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } } }
I20260806 03:46:23.254513 20121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:23.254928 20133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:23.255172 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:23.243735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:41839 (local address 127.0.0.1:33983)
0806 03:46:23.243952 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:23.243962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:23.244000 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:46:23.244726 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:23.244734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:23.244788 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:23.244940 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:23.245606 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:23.246511 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:23.247657 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:23.248114 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:23.248273 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:23.251112 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:23.251116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:23.251116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:23.251129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:23.253155 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:23.253641 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:23.253645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:23.253645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:23.253774 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:23.254062 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:23.254063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:23.254063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:23.254418 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:23.254583 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:23.254862 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 03:46:23.254965 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260806 03:46:23.256206 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:23.256521 20133 catalog_manager.cc:1347] Generated new cluster ID: 2b7a026cfdb84acdb3465bf09922d39c
I20260806 03:46:23.256567 20133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:23.265156 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:23.257893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:59765 (local address 127.0.0.1:33983)
0806 03:46:23.257927 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:23.257929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:23.257937 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:23.257958 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:23.257959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:23.257968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:23.258034 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:23.258615 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:23.259370 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:23.260338 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:23.260736 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:23.260806 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:23.262196 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:23.262197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:23.262198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:23.262205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:23.263852 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:23.264287 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:23.264289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:23.264289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:23.264353 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:23.264602 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:23.264603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:23.264603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:23.264748 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:23.264857 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:23.264979 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 03:46:23.265063 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:23.267940 20133 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:23.268494 20133 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:23.278690 20133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846079a6361f4f10af4995b3c4fff1b9: Generated new TSK 0
I20260806 03:46:23.278908 20133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:23.284456 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20048
W20260806 03:46:23.290407 20033 connection.cc:535] client connection to 127.0.0.1:33983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33983): Connection reset by peer (error 104)
2026-08-06T03:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:23 dist-test-slave-10pc krb5kdc[20147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:23 dist-test-slave-10pc krb5kdc[20147](info): set up 2 sockets
Aug 06 03:46:23 dist-test-slave-10pc krb5kdc[20147](info): commencing operation
krb5kdc: starting...
W20260806 03:46:24.333973 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:24 dist-test-slave-10pc krb5kdc[20147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:24.468248 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:24.554873 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:24.555142 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:24.555186 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:24.556344 20160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:24.556417 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:24.556442 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:24.556464 20160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:24.556516 20160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:24.558056 20160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:24.558355 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:24.558605 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:24.563787 20168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:24.563807 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:24.563859 20160 server_base.cc:1034] running on GCE node
W20260806 03:46:24.563799 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:24.564333 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:24.564740 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:24.565943 20160 hybrid_clock.cc:648] HybridClock initialized: now 1785987984565916 us; error 36 us; skew 500 ppm
Aug 06 03:46:24 dist-test-slave-10pc krb5kdc[20147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:24.569031 20160 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:24.569530 20160 webserver.cc:460] Webserver started at http://127.0.0.1:38997/ using document root <none> and password file <none>
I20260806 03:46:24.569795 20160 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:24.569842 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:24.570013 20160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:24.571126 20160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "c2afeb0796fa4c26ac8bf215ef21ce66"
format_stamp: "Formatted at 2026-08-06 03:46:24 on dist-test-slave-10pc"
server_key: "1a3fa98e3acb31c5b1190d646df385cc"
server_key_iv: "523149e5081f66bd3a0302558e4346cb"
server_key_version: "encryptionkey@0"
I20260806 03:46:24.571489 20160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "c2afeb0796fa4c26ac8bf215ef21ce66"
format_stamp: "Formatted at 2026-08-06 03:46:24 on dist-test-slave-10pc"
server_key: "1a3fa98e3acb31c5b1190d646df385cc"
server_key_iv: "523149e5081f66bd3a0302558e4346cb"
server_key_version: "encryptionkey@0"
I20260806 03:46:24.572970 20160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:24.574018 20175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:24.574222 20160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:24.574326 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "c2afeb0796fa4c26ac8bf215ef21ce66"
format_stamp: "Formatted at 2026-08-06 03:46:24 on dist-test-slave-10pc"
server_key: "1a3fa98e3acb31c5b1190d646df385cc"
server_key_iv: "523149e5081f66bd3a0302558e4346cb"
server_key_version: "encryptionkey@0"
I20260806 03:46:24.574416 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:24.584947 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:24.586854 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:24.587030 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:24.591332 20160 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38155
I20260806 03:46:24.591367 20231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38155 every 8 connection(s)
I20260806 03:46:24.591754 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:24.592741 20232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:24.594125 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20160
I20260806 03:46:24.594234 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:24.594435 30816 external_mini_cluster.cc:1428] Setting key 301583a410e11bef9b33274e47d9afe6
I20260806 03:46:24.595351 20232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66: Bootstrap starting.
I20260806 03:46:24.596032 20232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:24.596362 20232 log.cc:824] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:24.597098 20232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66: No bootstrap required, opened a new log
I20260806 03:46:24.599511 20232 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38155 } }
I20260806 03:46:24.599714 20232 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:24.599747 20232 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2afeb0796fa4c26ac8bf215ef21ce66, State: Initialized, Role: FOLLOWER
I20260806 03:46:24.599902 20232 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38155 } }
I20260806 03:46:24.599974 20232 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:24.599996 20232 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:24.600061 20232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:24.600565 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:24.595004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:36163 (local address 127.0.0.1:38155)
0806 03:46:24.595174 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:24.595182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:24.595207 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:46:24.595684 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:24.595691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:24.595741 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:24.595865 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:24.596160 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:24.597329 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:24.598422 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:24.598814 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:24.598889 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:24.599703 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:24.599726 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:24.599726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:24.599965 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:24.600024 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:24.600039 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:24.600123 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:24.600269 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 03:46:24.600305 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260806 03:46:24.600790 20232 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38155 } }
I20260806 03:46:24.600903 20232 leader_election.cc:302] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2afeb0796fa4c26ac8bf215ef21ce66; no voters:
I20260806 03:46:24.601063 20232 leader_election.cc:288] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:24.601099 20237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:24.601311 20232 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:24.601598 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:24.601353 20237 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [term 1 LEADER]: Becoming Leader. State: Replica: c2afeb0796fa4c26ac8bf215ef21ce66, State: Running, Role: LEADER
I20260806 03:46:24.602782 20237 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38155 } }
I20260806 03:46:24.603421 20238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38155 } } }
I20260806 03:46:24.603421 20239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2afeb0796fa4c26ac8bf215ef21ce66. Latest consensus state: current_term: 1 leader_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2afeb0796fa4c26ac8bf215ef21ce66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38155 } } }
I20260806 03:46:24.603535 20238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:24.603541 20239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66 [sys.catalog]: This master's current role is: LEADER
W20260806 03:46:24.604231 20258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:24.604298 20258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:24.604393 20256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:24.604598 20256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:24.606426 20256 catalog_manager.cc:1347] Generated new cluster ID: 337bed9082ed4ca68b5aa9a74ccc5e7c
I20260806 03:46:24.606482 20256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:24.606775 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:24.603036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:38843 (local address 127.0.0.1:38155)
0806 03:46:24.603078 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:24.603080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:24.603288 (+ 208us) server_negotiation.cc:407] Connection header received
0806 03:46:24.603310 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:24.603311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:24.603321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:24.603366 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:24.603566 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:24.604342 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:24.605413 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:24.605699 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:24.605763 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:24.606512 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:24.606513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:24.606513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:24.606517 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:24.606529 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:24.606533 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:24.606602 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:24.606671 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:46:24.606679 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:46:24.622889 20256 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:24.623453 20256 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:24.633742 20256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2afeb0796fa4c26ac8bf215ef21ce66: Generated new TSK 0
I20260806 03:46:24.633951 20256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:24.638141 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20160
W20260806 03:46:24.644032 20145 connection.cc:535] client connection to 127.0.0.1:38155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38155): Connection reset by peer (error 104)
2026-08-06T03:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:24 dist-test-slave-10pc krb5kdc[20263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:24 dist-test-slave-10pc krb5kdc[20263](info): set up 2 sockets
Aug 06 03:46:24 dist-test-slave-10pc krb5kdc[20263](info): commencing operation
krb5kdc: starting...
W20260806 03:46:25.687294 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:25 dist-test-slave-10pc krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:46:25.813848 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:25.897991 20276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:25.898231 20276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:25.898270 20276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:25.899410 20276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:25.899475 20276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:25.899502 20276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:25.899531 20276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:25.899555 20276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:25.901114 20276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:25.901383 20276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:25.901603 20276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:25.906512 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:25.906571 20283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:25.906484 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:25.906687 20276 server_base.cc:1034] running on GCE node
I20260806 03:46:25.906963 20276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:25.907203 20276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:25.908344 20276 hybrid_clock.cc:648] HybridClock initialized: now 1785987985908325 us; error 41 us; skew 500 ppm
Aug 06 03:46:25 dist-test-slave-10pc krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:25.910892 20276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:46:25.911387 20276 webserver.cc:460] Webserver started at http://127.0.0.1:37331/ using document root <none> and password file <none>
I20260806 03:46:25.911615 20276 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:25.911687 20276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:25.911844 20276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:25.912968 20276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "470a6581b51b4cfa910e9a7978ece875"
format_stamp: "Formatted at 2026-08-06 03:46:25 on dist-test-slave-10pc"
server_key: "cec7dd32250b99c5a886a732b43e177d"
server_key_iv: "eee1d5b52e6fb6ce9073784df3d464a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:25.913352 20276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "470a6581b51b4cfa910e9a7978ece875"
format_stamp: "Formatted at 2026-08-06 03:46:25 on dist-test-slave-10pc"
server_key: "cec7dd32250b99c5a886a732b43e177d"
server_key_iv: "eee1d5b52e6fb6ce9073784df3d464a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:25.914721 20276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:25.915488 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:25.915681 20276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:25.915791 20276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "470a6581b51b4cfa910e9a7978ece875"
format_stamp: "Formatted at 2026-08-06 03:46:25 on dist-test-slave-10pc"
server_key: "cec7dd32250b99c5a886a732b43e177d"
server_key_iv: "eee1d5b52e6fb6ce9073784df3d464a2"
server_key_version: "encryptionkey@0"
I20260806 03:46:25.915867 20276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:25.919293 20276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:25.921686 20276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:25.921857 20276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:25.925874 20276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37865
I20260806 03:46:25.925915 20345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37865 every 8 connection(s)
I20260806 03:46:25.926283 20276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:25.927477 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:25.929381 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20276
I20260806 03:46:25.929636 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:25.929847 30816 external_mini_cluster.cc:1428] Setting key e4edf7180f21b3ef82ac8d189e143d57
I20260806 03:46:25.930536 20346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875: Bootstrap starting.
I20260806 03:46:25.931264 20346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:25.931633 20346 log.cc:824] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:25.932399 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:25.930412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:54101 (local address 127.0.0.1:37865)
0806 03:46:25.930583 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:25.930591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:25.930646 (+ 55us) server_negotiation.cc:407] Connection header received
0806 03:46:25.931078 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:25.931085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:25.931111 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:25.931228 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:25.931975 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:25.931978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:25.931979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:25.931983 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:25.932034 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:25.932046 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:25.932158 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:25.932198 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 03:46:25.932241 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260806 03:46:25.932559 20346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875: No bootstrap required, opened a new log
I20260806 03:46:25.933458 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:25.935081 20346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470a6581b51b4cfa910e9a7978ece875" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37865 } }
I20260806 03:46:25.935295 20346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:25.935339 20346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470a6581b51b4cfa910e9a7978ece875, State: Initialized, Role: FOLLOWER
I20260806 03:46:25.935489 20346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470a6581b51b4cfa910e9a7978ece875" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37865 } }
I20260806 03:46:25.935609 20346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:25.935664 20346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:25.935722 20346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:25.935961 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:25.934864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:56523 (local address 127.0.0.1:37865)
0806 03:46:25.934901 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:25.934903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:25.934983 (+ 80us) server_negotiation.cc:407] Connection header received
0806 03:46:25.935050 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:25.935050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:25.935054 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:25.935105 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:25.935672 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:25.935674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:25.935674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:25.935678 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:25.935693 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:25.935697 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:25.935751 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:25.935883 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 03:46:25.935898 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:46:25.936494 20346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470a6581b51b4cfa910e9a7978ece875" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37865 } }
I20260806 03:46:25.936640 20346 leader_election.cc:302] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 470a6581b51b4cfa910e9a7978ece875; no voters:
I20260806 03:46:25.936841 20346 leader_election.cc:288] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:25.936896 20356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:25.937067 20356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [term 1 LEADER]: Becoming Leader. State: Replica: 470a6581b51b4cfa910e9a7978ece875, State: Running, Role: LEADER
I20260806 03:46:25.937132 20346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:25.937207 20356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470a6581b51b4cfa910e9a7978ece875" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37865 } }
I20260806 03:46:25.937750 20358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "470a6581b51b4cfa910e9a7978ece875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470a6581b51b4cfa910e9a7978ece875" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37865 } } }
I20260806 03:46:25.937876 20358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:25.938416 20359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 470a6581b51b4cfa910e9a7978ece875. Latest consensus state: current_term: 1 leader_uuid: "470a6581b51b4cfa910e9a7978ece875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470a6581b51b4cfa910e9a7978ece875" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37865 } } }
I20260806 03:46:25.938496 20359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875 [sys.catalog]: This master's current role is: LEADER
W20260806 03:46:25.939116 20373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:25.939177 20373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:25.939236 20366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:25.939375 20366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:25.940856 20366 catalog_manager.cc:1347] Generated new cluster ID: b2b57a1bd97742b1b609785a1a14e49f
I20260806 03:46:25.940899 20366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:25.950999 20366 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:25.951176 20366 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:25.956125 20366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 470a6581b51b4cfa910e9a7978ece875: Generated new TSK 0
I20260806 03:46:25.956326 20366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:25.964627 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20276
2026-08-06T03:46:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:26 dist-test-slave-10pc krb5kdc[20378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:26 dist-test-slave-10pc krb5kdc[20378](info): set up 2 sockets
Aug 06 03:46:26 dist-test-slave-10pc krb5kdc[20378](info): commencing operation
krb5kdc: starting...
W20260806 03:46:27.021976 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:27 dist-test-slave-10pc krb5kdc[20378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:27.147372 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35993
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:35993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:27.236763 20391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:27.236979 20391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:27.237002 20391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:27.238384 20391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:27.238441 20391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:27.238457 20391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:27.238471 20391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:27.238482 20391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:27.239981 20391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:35993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:35993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:27.240181 20391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:27.240363 20391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:27.245369 20397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:27.245455 20396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:27.245453 20399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:27.245769 20391 server_base.cc:1034] running on GCE node
I20260806 03:46:27.246025 20391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:27.246345 20391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:27.247546 20391 hybrid_clock.cc:648] HybridClock initialized: now 1785987987247528 us; error 49 us; skew 500 ppm
Aug 06 03:46:27 dist-test-slave-10pc krb5kdc[20378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987987, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:27.255462 20391 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:27.255952 20391 webserver.cc:460] Webserver started at http://127.30.24.62:46311/ using document root <none> and password file <none>
I20260806 03:46:27.256150 20391 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:27.256193 20391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:27.256287 20391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:27.257421 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "c7ba6d83f20842588d02a3985cac2afb"
format_stamp: "Formatted at 2026-08-06 03:46:27 on dist-test-slave-10pc"
server_key: "0f58f37a7750f134ee11e4bed4b515df"
server_key_iv: "c603f9a4ba23b69e49cde941a1345e08"
server_key_version: "encryptionkey@0"
I20260806 03:46:27.257751 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "c7ba6d83f20842588d02a3985cac2afb"
format_stamp: "Formatted at 2026-08-06 03:46:27 on dist-test-slave-10pc"
server_key: "0f58f37a7750f134ee11e4bed4b515df"
server_key_iv: "c603f9a4ba23b69e49cde941a1345e08"
server_key_version: "encryptionkey@0"
I20260806 03:46:27.259074 20391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:27.259784 20406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:27.259959 20391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 03:46:27.260048 20391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "c7ba6d83f20842588d02a3985cac2afb"
format_stamp: "Formatted at 2026-08-06 03:46:27 on dist-test-slave-10pc"
server_key: "0f58f37a7750f134ee11e4bed4b515df"
server_key_iv: "c603f9a4ba23b69e49cde941a1345e08"
server_key_version: "encryptionkey@0"
I20260806 03:46:27.260104 20391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:27.285168 20391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:27.287081 20391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:27.287192 20391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:27.291052 20391 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:35993
I20260806 03:46:27.291422 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:35993 every 8 connection(s)
I20260806 03:46:27.291495 20391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:27.292204 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:27.292970 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20391
I20260806 03:46:27.293066 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:27.293278 30816 external_mini_cluster.cc:1428] Setting key 2572d9505d7adb1ec43bce94fe9f3ff5
I20260806 03:46:27.295466 20460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb: Bootstrap starting.
I20260806 03:46:27.296231 20460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:27.296588 20460 log.cc:824] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:27.297308 20460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb: No bootstrap required, opened a new log
I20260806 03:46:27.299536 20460 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ba6d83f20842588d02a3985cac2afb" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35993 } }
I20260806 03:46:27.299674 20460 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:27.299700 20460 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ba6d83f20842588d02a3985cac2afb, State: Initialized, Role: FOLLOWER
I20260806 03:46:27.299844 20460 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ba6d83f20842588d02a3985cac2afb" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35993 } }
I20260806 03:46:27.299911 20460 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:27.299933 20460 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:27.300000 20460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 0 FOLLOWER]: Advancing to term 1
Aug 06 03:46:27 dist-test-slave-10pc krb5kdc[20378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:27.300640 20460 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ba6d83f20842588d02a3985cac2afb" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35993 } }
I20260806 03:46:27.300773 20460 leader_election.cc:302] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7ba6d83f20842588d02a3985cac2afb; no voters:
I20260806 03:46:27.300959 20460 leader_election.cc:288] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:27.301002 20465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:27.301169 20465 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [term 1 LEADER]: Becoming Leader. State: Replica: c7ba6d83f20842588d02a3985cac2afb, State: Running, Role: LEADER
I20260806 03:46:27.301307 20460 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:27.301307 20465 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ba6d83f20842588d02a3985cac2afb" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35993 } }
I20260806 03:46:27.301793 20466 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7ba6d83f20842588d02a3985cac2afb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ba6d83f20842588d02a3985cac2afb" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35993 } } }
I20260806 03:46:27.301880 20466 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:27.301985 20467 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7ba6d83f20842588d02a3985cac2afb. Latest consensus state: current_term: 1 leader_uuid: "c7ba6d83f20842588d02a3985cac2afb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ba6d83f20842588d02a3985cac2afb" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 35993 } } }
I20260806 03:46:27.302264 20467 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:27.302511 20474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:27.303025 20474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:27.304903 20474 catalog_manager.cc:1347] Generated new cluster ID: 3f083e5d389940d9920684a88716af87
I20260806 03:46:27.304967 20474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:27.305414 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:27.293838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49209 (local address 127.30.24.62:35993)
0806 03:46:27.294043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:27.294054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:27.294090 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:46:27.294663 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:27.294673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:27.294749 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:27.294917 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:27.295572 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:27.296969 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:27.297807 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:27.298433 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:27.298548 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:27.301076 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:27.301081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:27.301081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:27.301093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:27.303230 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:27.303717 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:27.303721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:27.303721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:27.303831 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:27.304114 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:27.304116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:27.304116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:27.304429 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:27.304617 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:27.305104 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 03:46:27.305215 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260806 03:46:27.306389 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:27.315490 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:27.308002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42023 (local address 127.30.24.62:35993)
0806 03:46:27.308039 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:27.308041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:27.308182 (+ 141us) server_negotiation.cc:407] Connection header received
0806 03:46:27.308228 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:27.308229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:27.308240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:27.308295 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:27.308908 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:27.309967 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:27.310927 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:27.311249 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:27.311326 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:27.312672 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:27.312674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:27.312674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:27.312682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:27.314255 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:27.314670 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:27.314671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:27.314671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:27.314733 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:27.314966 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:27.314967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:27.314967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:27.315097 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:27.315201 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:27.315317 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 03:46:27.315383 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:27.323910 20474 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:27.324671 20474 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:27.333521 20474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7ba6d83f20842588d02a3985cac2afb: Generated new TSK 0
I20260806 03:46:27.333732 20474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:27.339032 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20391
W20260806 03:46:27.345487 20376 connection.cc:535] client connection to 127.30.24.62:35993 recv error: Network error: failed to read from TLS socket (remote: 127.30.24.62:35993): Connection reset by peer (error 104)
2026-08-06T03:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:27 dist-test-slave-10pc krb5kdc[20491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:27 dist-test-slave-10pc krb5kdc[20491](info): set up 2 sockets
Aug 06 03:46:27 dist-test-slave-10pc krb5kdc[20491](info): commencing operation
krb5kdc: starting...
W20260806 03:46:28.473567 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:28 dist-test-slave-10pc krb5kdc[20491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:28.599730 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37213
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:28.683736 20505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:28.683949 20505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:28.683972 20505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:28.685171 20505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:28.685218 20505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:28.685232 20505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:28.685245 20505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:28.685257 20505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:28.686754 20505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:37213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:28.687006 20505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:28.687244 20505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:28.692287 20511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:28.692283 20510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:28.692317 20505 server_base.cc:1034] running on GCE node
W20260806 03:46:28.692301 20513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:28.692796 20505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:28.693100 20505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:28.694274 20505 hybrid_clock.cc:648] HybridClock initialized: now 1785987988694254 us; error 39 us; skew 500 ppm
Aug 06 03:46:28 dist-test-slave-10pc krb5kdc[20491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987988, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:28.697232 20505 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:28.697672 20505 webserver.cc:460] Webserver started at http://127.30.24.62:33733/ using document root <none> and password file <none>
I20260806 03:46:28.697902 20505 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:28.697973 20505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:28.698135 20505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:28.699216 20505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "9785ec64c621472ba7240ea8c2774356"
format_stamp: "Formatted at 2026-08-06 03:46:28 on dist-test-slave-10pc"
server_key: "590e62f6cc6f122d475c6852a744498f"
server_key_iv: "289bb824042f757a0189e967167456b8"
server_key_version: "encryptionkey@0"
I20260806 03:46:28.699602 20505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "9785ec64c621472ba7240ea8c2774356"
format_stamp: "Formatted at 2026-08-06 03:46:28 on dist-test-slave-10pc"
server_key: "590e62f6cc6f122d475c6852a744498f"
server_key_iv: "289bb824042f757a0189e967167456b8"
server_key_version: "encryptionkey@0"
I20260806 03:46:28.701113 20505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:46:28.701956 20520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:28.702169 20505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:28.702281 20505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "9785ec64c621472ba7240ea8c2774356"
format_stamp: "Formatted at 2026-08-06 03:46:28 on dist-test-slave-10pc"
server_key: "590e62f6cc6f122d475c6852a744498f"
server_key_iv: "289bb824042f757a0189e967167456b8"
server_key_version: "encryptionkey@0"
I20260806 03:46:28.702358 20505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:28.711467 20505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:28.713887 20505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:28.714061 20505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:28.718051 20505 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:37213
I20260806 03:46:28.718191 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:37213 every 8 connection(s)
I20260806 03:46:28.718557 20505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:28.719434 20574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:28.721836 20574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356: Bootstrap starting.
I20260806 03:46:28.722424 20574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:28.722711 20574 log.cc:824] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:28.723414 20574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356: No bootstrap required, opened a new log
I20260806 03:46:28.725126 20574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9785ec64c621472ba7240ea8c2774356" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37213 } }
I20260806 03:46:28.725284 20574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:28.725308 20574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9785ec64c621472ba7240ea8c2774356, State: Initialized, Role: FOLLOWER
I20260806 03:46:28.725425 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20505
I20260806 03:46:28.725497 20574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9785ec64c621472ba7240ea8c2774356" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37213 } }
I20260806 03:46:28.725589 20574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:28.725576 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:28.725630 20574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:28.725690 20574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:28.725804 30816 external_mini_cluster.cc:1428] Setting key 732448dce64538076d7642788d6e63a5
I20260806 03:46:28.726768 20574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9785ec64c621472ba7240ea8c2774356" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37213 } }
I20260806 03:46:28.726907 20574 leader_election.cc:302] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9785ec64c621472ba7240ea8c2774356; no voters:
I20260806 03:46:28.727092 20574 leader_election.cc:288] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:28.727121 20579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:28.727262 20579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [term 1 LEADER]: Becoming Leader. State: Replica: 9785ec64c621472ba7240ea8c2774356, State: Running, Role: LEADER
I20260806 03:46:28.727388 20574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:28.727407 20579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9785ec64c621472ba7240ea8c2774356" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37213 } }
I20260806 03:46:28.727813 20579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9785ec64c621472ba7240ea8c2774356. Latest consensus state: current_term: 1 leader_uuid: "9785ec64c621472ba7240ea8c2774356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9785ec64c621472ba7240ea8c2774356" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37213 } } }
I20260806 03:46:28.727825 20580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9785ec64c621472ba7240ea8c2774356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9785ec64c621472ba7240ea8c2774356" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37213 } } }
I20260806 03:46:28.727965 20580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:28.728235 20579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:28.728835 20586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:28.729358 20586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:28.731916 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:28.726341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32985 (local address 127.30.24.62:37213)
0806 03:46:28.726543 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:28.726551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:28.726576 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:46:28.726672 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:28.726677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:28.726723 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:28.726829 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:28.727101 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:28.728577 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:28.729707 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:28.730315 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:28.730455 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:28.731222 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:28.731226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:28.731226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:28.731447 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:28.731484 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:28.731500 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:28.731592 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:28.731668 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 03:46:28.731696 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260806 03:46:28.732141 20586 catalog_manager.cc:1347] Generated new cluster ID: 77bcc9b66fc5458d99197fd923ba40f5
I20260806 03:46:28.732213 20586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:28.736967 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:28.740839 20586 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:28.741339 20586 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:28.742262 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:28.738516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56839 (local address 127.30.24.62:37213)
0806 03:46:28.738553 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:28.738555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:28.738695 (+ 140us) server_negotiation.cc:407] Connection header received
0806 03:46:28.738730 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:28.738731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:28.738754 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:28.738803 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:28.738945 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:28.739676 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:28.740764 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:28.741096 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:28.741146 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:28.741975 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:28.741976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:28.741976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:28.741981 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:28.742009 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:28.742013 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:28.742071 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:28.742174 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 03:46:28.742187 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 03:46:28.749390 20586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9785ec64c621472ba7240ea8c2774356: Generated new TSK 0
I20260806 03:46:28.749584 20586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:28.753664 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20505
W20260806 03:46:28.760022 20489 connection.cc:535] client connection to 127.30.24.62:37213 recv error: Network error: failed to read from TLS socket (remote: 127.30.24.62:37213): Connection reset by peer (error 104)
2026-08-06T03:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:28 dist-test-slave-10pc krb5kdc[20605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:28 dist-test-slave-10pc krb5kdc[20605](info): set up 2 sockets
Aug 06 03:46:28 dist-test-slave-10pc krb5kdc[20605](info): commencing operation
krb5kdc: starting...
W20260806 03:46:29.817791 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:29 dist-test-slave-10pc krb5kdc[20605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:29.940827 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37347
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:30.027878 20618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:30.028138 20618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:30.028168 20618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:30.029570 20618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:30.029629 20618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:30.029646 20618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:30.029661 20618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:30.029675 20618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:30.031404 20618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:37347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:30.031704 20618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:30.031982 20618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:30.037834 20626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:30.037822 20624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:30.037793 20623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:30.038081 20618 server_base.cc:1034] running on GCE node
I20260806 03:46:30.038555 20618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:30.038859 20618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:30.040088 20618 hybrid_clock.cc:648] HybridClock initialized: now 1785987990040048 us; error 88 us; skew 500 ppm
Aug 06 03:46:30 dist-test-slave-10pc krb5kdc[20605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987990, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:30.043002 20618 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:30.043552 20618 webserver.cc:460] Webserver started at http://127.30.24.62:41331/ using document root <none> and password file <none>
I20260806 03:46:30.043812 20618 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:30.043859 20618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:30.043962 20618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:30.045197 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "9a29b7abca3d4f7590a795efeefc2314"
format_stamp: "Formatted at 2026-08-06 03:46:30 on dist-test-slave-10pc"
server_key: "c234a8c48369ae8089a392d0d0c0a255"
server_key_iv: "98f3e8b8a63b0648d6accbcfc30abb1b"
server_key_version: "encryptionkey@0"
I20260806 03:46:30.045544 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "9a29b7abca3d4f7590a795efeefc2314"
format_stamp: "Formatted at 2026-08-06 03:46:30 on dist-test-slave-10pc"
server_key: "c234a8c48369ae8089a392d0d0c0a255"
server_key_iv: "98f3e8b8a63b0648d6accbcfc30abb1b"
server_key_version: "encryptionkey@0"
I20260806 03:46:30.046965 20618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 03:46:30.047834 20633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:30.048059 20618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:30.048200 20618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "9a29b7abca3d4f7590a795efeefc2314"
format_stamp: "Formatted at 2026-08-06 03:46:30 on dist-test-slave-10pc"
server_key: "c234a8c48369ae8089a392d0d0c0a255"
server_key_iv: "98f3e8b8a63b0648d6accbcfc30abb1b"
server_key_version: "encryptionkey@0"
I20260806 03:46:30.048271 20618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:30.052232 20618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:30.054837 20618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:30.054957 20618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:30.059422 20618 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:37347
I20260806 03:46:30.059461 20686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:37347 every 8 connection(s)
I20260806 03:46:30.059868 20618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:30.061081 20687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:30.063738 20687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314: Bootstrap starting.
I20260806 03:46:30.064368 20687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:30.064754 20687 log.cc:824] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:30.065548 20687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314: No bootstrap required, opened a new log
I20260806 03:46:30.066685 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20618
I20260806 03:46:30.066792 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:30.067071 30816 external_mini_cluster.cc:1428] Setting key e81e82eea94384aaa389b8fafaea887f
I20260806 03:46:30.067890 20687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a29b7abca3d4f7590a795efeefc2314" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37347 } }
I20260806 03:46:30.068145 20687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:30.068187 20687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a29b7abca3d4f7590a795efeefc2314, State: Initialized, Role: FOLLOWER
I20260806 03:46:30.068332 20687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a29b7abca3d4f7590a795efeefc2314" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37347 } }
I20260806 03:46:30.068434 20687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:30.068542 20687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:30.068616 20687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:30.069802 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:30.067759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41391 (local address 127.30.24.62:37347)
0806 03:46:30.068011 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:30.068022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:30.068061 (+ 39us) server_negotiation.cc:407] Connection header received
0806 03:46:30.068196 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:30.068206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:30.068242 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:30.068416 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:30.069229 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:30.069233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:30.069233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:30.069238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:30.069291 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:30.069309 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:30.069398 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:30.069464 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 03:46:30.069520 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":127,"threads_started":1}
I20260806 03:46:30.070232 20687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a29b7abca3d4f7590a795efeefc2314" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37347 } }
I20260806 03:46:30.070380 20687 leader_election.cc:302] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a29b7abca3d4f7590a795efeefc2314; no voters:
I20260806 03:46:30.070632 20687 leader_election.cc:288] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:30.070670 20692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:30.070797 20692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [term 1 LEADER]: Becoming Leader. State: Replica: 9a29b7abca3d4f7590a795efeefc2314, State: Running, Role: LEADER
I20260806 03:46:30.070891 20692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a29b7abca3d4f7590a795efeefc2314" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37347 } }
I20260806 03:46:30.070915 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:30.070941 20687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:30.072372 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a29b7abca3d4f7590a795efeefc2314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a29b7abca3d4f7590a795efeefc2314" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37347 } } }
I20260806 03:46:30.072559 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:30.072592 20694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a29b7abca3d4f7590a795efeefc2314. Latest consensus state: current_term: 1 leader_uuid: "9a29b7abca3d4f7590a795efeefc2314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a29b7abca3d4f7590a795efeefc2314" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 37347 } } }
I20260806 03:46:30.073185 20694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314 [sys.catalog]: This master's current role is: LEADER
W20260806 03:46:30.073239 20711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:30.073305 20711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:30.073392 20712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:30.073609 20712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:30.075440 20712 catalog_manager.cc:1347] Generated new cluster ID: 753371c014734bf5beaa76dc2dfb3949
I20260806 03:46:30.075512 20712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:30.075531 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:30.073061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37849 (local address 127.30.24.62:37347)
0806 03:46:30.073125 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:30.073127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:30.073365 (+ 238us) server_negotiation.cc:407] Connection header received
0806 03:46:30.074379 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:30.074380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:30.074386 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:30.074415 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:30.075175 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:30.075177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:30.075177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:30.075182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:30.075197 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:30.075202 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:30.075233 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:30.075384 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 03:46:30.075404 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 03:46:30.081842 20712 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:30.082104 20712 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:30.087949 20712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a29b7abca3d4f7590a795efeefc2314: Generated new TSK 0
I20260806 03:46:30.088201 20712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:30.095314 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20618
2026-08-06T03:46:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:30 dist-test-slave-10pc krb5kdc[20718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:30 dist-test-slave-10pc krb5kdc[20718](info): set up 2 sockets
Aug 06 03:46:30 dist-test-slave-10pc krb5kdc[20718](info): commencing operation
krb5kdc: starting...
W20260806 03:46:31.149785 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:31 dist-test-slave-10pc krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:31.283135 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33049
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:33049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:31.379417 20731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:31.379648 20731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:31.379675 20731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:31.380963 20731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:31.381006 20731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:31.381021 20731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:31.381039 20731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:31.381053 20731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:31.382683 20731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:33049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:33049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:31.382903 20731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:31.383100 20731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:31.388651 20736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:31.388648 20739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:31.388644 20737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:31.388669 20731 server_base.cc:1034] running on GCE node
I20260806 03:46:31.389163 20731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:31.389497 20731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:31.390674 20731 hybrid_clock.cc:648] HybridClock initialized: now 1785987991390652 us; error 40 us; skew 500 ppm
Aug 06 03:46:31 dist-test-slave-10pc krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987991, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:31.393360 20731 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:31.393886 20731 webserver.cc:460] Webserver started at http://127.30.24.62:41183/ using document root <none> and password file <none>
I20260806 03:46:31.394147 20731 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:31.394220 20731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:31.394375 20731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:31.395476 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "0759da7e9cbc4b058b717895472a0925"
format_stamp: "Formatted at 2026-08-06 03:46:31 on dist-test-slave-10pc"
server_key: "f2759ebccc89b490c94f5742a1f0c9d9"
server_key_iv: "8d80497560938d790005d177ae9cf9db"
server_key_version: "encryptionkey@0"
I20260806 03:46:31.395848 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "0759da7e9cbc4b058b717895472a0925"
format_stamp: "Formatted at 2026-08-06 03:46:31 on dist-test-slave-10pc"
server_key: "f2759ebccc89b490c94f5742a1f0c9d9"
server_key_iv: "8d80497560938d790005d177ae9cf9db"
server_key_version: "encryptionkey@0"
I20260806 03:46:31.397526 20731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:31.398344 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:31.398540 20731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:31.398656 20731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "0759da7e9cbc4b058b717895472a0925"
format_stamp: "Formatted at 2026-08-06 03:46:31 on dist-test-slave-10pc"
server_key: "f2759ebccc89b490c94f5742a1f0c9d9"
server_key_iv: "8d80497560938d790005d177ae9cf9db"
server_key_version: "encryptionkey@0"
I20260806 03:46:31.398744 20731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:31.412376 20731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:31.414635 20731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:31.414810 20731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:31.419169 20731 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:33049
I20260806 03:46:31.419211 20798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:33049 every 8 connection(s)
I20260806 03:46:31.419620 20731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:31.420605 20799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:31.422935 20799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925: Bootstrap starting.
I20260806 03:46:31.423555 20799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:31.423858 20799 log.cc:824] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:31.424647 20799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925: No bootstrap required, opened a new log
I20260806 03:46:31.426410 20799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0759da7e9cbc4b058b717895472a0925" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33049 } }
I20260806 03:46:31.426668 20799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:31.426771 20799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0759da7e9cbc4b058b717895472a0925, State: Initialized, Role: FOLLOWER
I20260806 03:46:31.426941 20799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0759da7e9cbc4b058b717895472a0925" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33049 } }
I20260806 03:46:31.427016 20799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:31.427073 20799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:31.427129 20799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:31.428056 20799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0759da7e9cbc4b058b717895472a0925" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33049 } }
I20260806 03:46:31.428180 20799 leader_election.cc:302] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0759da7e9cbc4b058b717895472a0925; no voters:
I20260806 03:46:31.428387 20799 leader_election.cc:288] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:31.428423 20802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:31.428618 20802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [term 1 LEADER]: Becoming Leader. State: Replica: 0759da7e9cbc4b058b717895472a0925, State: Running, Role: LEADER
I20260806 03:46:31.428691 20799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:31.428797 20802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0759da7e9cbc4b058b717895472a0925" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33049 } }
I20260806 03:46:31.429193 20803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0759da7e9cbc4b058b717895472a0925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0759da7e9cbc4b058b717895472a0925" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33049 } } }
I20260806 03:46:31.429280 20803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:31.429382 20804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0759da7e9cbc4b058b717895472a0925. Latest consensus state: current_term: 1 leader_uuid: "0759da7e9cbc4b058b717895472a0925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0759da7e9cbc4b058b717895472a0925" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 33049 } } }
I20260806 03:46:31.429641 20804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:31.429628 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20731
I20260806 03:46:31.429769 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:31.430029 30816 external_mini_cluster.cc:1428] Setting key d85fb496e6a39ebae3657d688bdae3f3
I20260806 03:46:31.430027 20811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:31.430637 20811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:31.432276 20811 catalog_manager.cc:1347] Generated new cluster ID: 64bcad0f88a146c8ab78a9de98c2bbca
I20260806 03:46:31.432331 20811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:31.436697 20811 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:31.437299 20811 catalog_manager.cc:1504] Loading token signing keys...
Aug 06 03:46:31 dist-test-slave-10pc krb5kdc[20718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:31.442584 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:31.430659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35827 (local address 127.30.24.62:33049)
0806 03:46:31.430848 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:31.430858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:31.430892 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:46:31.431015 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:31.431022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:31.431078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:31.431229 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:31.431978 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:31.433390 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:31.434501 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:31.435137 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:31.435249 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:31.438379 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:31.438384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:31.438384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:31.438397 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:31.440546 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:31.440962 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:31.440965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:31.440966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:31.441074 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:31.441394 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:31.441395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:31.441396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:31.441708 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:31.441865 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:31.442073 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 03:46:31.442349 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260806 03:46:31.443586 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:31.446651 20811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0759da7e9cbc4b058b717895472a0925: Generated new TSK 0
I20260806 03:46:31.446820 20811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:31.452268 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:31.445192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60881 (local address 127.30.24.62:33049)
0806 03:46:31.445277 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:31.445279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:31.445333 (+ 54us) server_negotiation.cc:407] Connection header received
0806 03:46:31.445412 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:31.445413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:31.445423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:31.445472 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:31.446103 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:31.446851 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:31.447924 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:31.448145 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:31.448206 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:31.449637 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:31.449638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:31.449639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:31.449645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:31.451127 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:31.451526 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:31.451527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:31.451528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:31.451583 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:31.451798 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:31.451798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:31.451799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:31.451901 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:31.451977 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:31.452127 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 03:46:31.452200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 03:46:31.454381 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20731
2026-08-06T03:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:31 dist-test-slave-10pc krb5kdc[20831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:31 dist-test-slave-10pc krb5kdc[20831](info): set up 2 sockets
Aug 06 03:46:31 dist-test-slave-10pc krb5kdc[20831](info): commencing operation
krb5kdc: starting...
W20260806 03:46:32.513491 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:32 dist-test-slave-10pc krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:32.639506 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:39791
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:39791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:32.722826 20844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:32.723091 20844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:32.723132 20844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:32.724306 20844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:32.724380 20844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:32.724404 20844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:32.724426 20844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:32.724489 20844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:32.726040 20844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:39791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:39791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:32.726354 20844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:32.726601 20844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:32.731491 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:32.731580 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:32.731487 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:32.731777 20844 server_base.cc:1034] running on GCE node
I20260806 03:46:32.732020 20844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:32.732317 20844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:32.733491 20844 hybrid_clock.cc:648] HybridClock initialized: now 1785987992733488 us; error 50 us; skew 500 ppm
Aug 06 03:46:32 dist-test-slave-10pc krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987992, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:32.741458 20844 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:32.741940 20844 webserver.cc:460] Webserver started at http://127.30.24.62:35561/ using document root <none> and password file <none>
I20260806 03:46:32.742190 20844 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:32.742237 20844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:32.742372 20844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:32.743482 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "f594344c52894f1ab8e574df53cbffaf"
format_stamp: "Formatted at 2026-08-06 03:46:32 on dist-test-slave-10pc"
server_key: "a56d9e55f80059a12fe1ab33cdea035d"
server_key_iv: "e2e54c1c1e1a7e6c3931885501bdb7c9"
server_key_version: "encryptionkey@0"
I20260806 03:46:32.743870 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "f594344c52894f1ab8e574df53cbffaf"
format_stamp: "Formatted at 2026-08-06 03:46:32 on dist-test-slave-10pc"
server_key: "a56d9e55f80059a12fe1ab33cdea035d"
server_key_iv: "e2e54c1c1e1a7e6c3931885501bdb7c9"
server_key_version: "encryptionkey@0"
I20260806 03:46:32.745558 20844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 03:46:32.746529 20859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:32.746811 20844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:32.746937 20844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "f594344c52894f1ab8e574df53cbffaf"
format_stamp: "Formatted at 2026-08-06 03:46:32 on dist-test-slave-10pc"
server_key: "a56d9e55f80059a12fe1ab33cdea035d"
server_key_iv: "e2e54c1c1e1a7e6c3931885501bdb7c9"
server_key_version: "encryptionkey@0"
I20260806 03:46:32.747071 20844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:32.791929 20844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:32.794811 20844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:32.795006 20844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:32.799685 20844 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:39791
I20260806 03:46:32.799752 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:39791 every 8 connection(s)
I20260806 03:46:32.800182 20844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:32.801214 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:32.803584 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf: Bootstrap starting.
I20260806 03:46:32.804150 20912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:32.804396 20912 log.cc:824] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:32.805236 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf: No bootstrap required, opened a new log
I20260806 03:46:32.806242 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20844
I20260806 03:46:32.806376 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:32.806578 30816 external_mini_cluster.cc:1428] Setting key 8f47b47fd22a738b05cb8119e7c02977
I20260806 03:46:32.807142 20912 raft_consensus.cc:348] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f594344c52894f1ab8e574df53cbffaf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39791 } }
I20260806 03:46:32.807345 20912 raft_consensus.cc:374] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:32.807422 20912 raft_consensus.cc:729] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f594344c52894f1ab8e574df53cbffaf, State: Initialized, Role: FOLLOWER
I20260806 03:46:32.807571 20912 consensus_queue.cc:260] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f594344c52894f1ab8e574df53cbffaf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39791 } }
I20260806 03:46:32.807638 20912 raft_consensus.cc:388] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:32.807682 20912 raft_consensus.cc:482] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:32.807735 20912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:32.808426 20912 raft_consensus.cc:504] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f594344c52894f1ab8e574df53cbffaf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39791 } }
I20260806 03:46:32.808607 20912 leader_election.cc:302] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f594344c52894f1ab8e574df53cbffaf; no voters:
I20260806 03:46:32.808816 20912 leader_election.cc:288] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:32.808854 20917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:32.809015 20917 raft_consensus.cc:686] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [term 1 LEADER]: Becoming Leader. State: Replica: f594344c52894f1ab8e574df53cbffaf, State: Running, Role: LEADER
I20260806 03:46:32.809093 20912 sys_catalog.cc:564] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:32.809213 20917 consensus_queue.cc:237] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f594344c52894f1ab8e574df53cbffaf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39791 } }
I20260806 03:46:32.809603 20917 sys_catalog.cc:455] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f594344c52894f1ab8e574df53cbffaf. Latest consensus state: current_term: 1 leader_uuid: "f594344c52894f1ab8e574df53cbffaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f594344c52894f1ab8e574df53cbffaf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39791 } } }
I20260806 03:46:32.809784 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f594344c52894f1ab8e574df53cbffaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f594344c52894f1ab8e574df53cbffaf" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39791 } } }
I20260806 03:46:32.809882 20917 sys_catalog.cc:458] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:32.809973 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf [sys.catalog]: This master's current role is: LEADER
W20260806 03:46:32.810981 20932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:32.811118 20932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:32.811223 20926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:32.811393 20926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:32.813215 20926 catalog_manager.cc:1347] Generated new cluster ID: eaf8468aaf54424e85d5f3bdc59622d6
I20260806 03:46:32.813270 20926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:32.813271 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:32.807149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39771 (local address 127.30.24.62:39791)
0806 03:46:32.807352 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:32.807362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:32.807397 (+ 35us) server_negotiation.cc:407] Connection header received
0806 03:46:32.807553 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:32.807560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:32.807621 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:32.807774 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:32.808026 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:32.809675 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:32.810872 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:32.811473 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:32.811578 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:32.812549 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:32.812553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:32.812554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:32.812762 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:32.812800 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:32.812815 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:32.812893 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:32.812972 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 03:46:32.813051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260806 03:46:32.814211 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:32.819484 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:32.815631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57393 (local address 127.30.24.62:39791)
0806 03:46:32.815699 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:32.815701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:32.815885 (+ 184us) server_negotiation.cc:407] Connection header received
0806 03:46:32.815916 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:32.815917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:32.815929 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:32.816002 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:32.816126 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:32.817206 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:32.818168 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:32.818466 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:32.818534 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:32.819234 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:32.819236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:32.819236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:32.819241 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:32.819256 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:32.819261 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:32.819320 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:32.819377 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 03:46:32.819400 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 03:46:32.825064 20926 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:32.825758 20926 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:32.831032 20926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f594344c52894f1ab8e574df53cbffaf: Generated new TSK 0
I20260806 03:46:32.831215 20926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:32.835039 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20844
2026-08-06T03:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:32 dist-test-slave-10pc krb5kdc[20943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:32 dist-test-slave-10pc krb5kdc[20943](info): set up 2 sockets
Aug 06 03:46:32 dist-test-slave-10pc krb5kdc[20943](info): commencing operation
krb5kdc: starting...
W20260806 03:46:33.909935 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:34 dist-test-slave-10pc krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:34.043424 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:39219
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:34.129366 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:34.129616 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:34.129652 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:34.130903 20956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:34.130976 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:34.130998 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:34.131022 20956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:34.131048 20956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:34.132566 20956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:34.132849 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:34.133095 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:34.138190 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:34.138226 20956 server_base.cc:1034] running on GCE node
W20260806 03:46:34.138155 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:34.138166 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:34.138738 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:34.139168 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:34.140383 20956 hybrid_clock.cc:648] HybridClock initialized: now 1785987994140352 us; error 40 us; skew 500 ppm
Aug 06 03:46:34 dist-test-slave-10pc krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987994, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:34.143128 20956 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:34.143631 20956 webserver.cc:460] Webserver started at http://127.30.24.62:39325/ using document root <none> and password file <none>
I20260806 03:46:34.143891 20956 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:34.143965 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:34.144124 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:34.145310 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "56a6d526310d478cb4fce34a44f0d197"
format_stamp: "Formatted at 2026-08-06 03:46:34 on dist-test-slave-10pc"
server_key: "2ba0cecd43c4ba11648394aeb9bb4567"
server_key_iv: "37b41625b7517967b8be8d40036e9a19"
server_key_version: "encryptionkey@0"
I20260806 03:46:34.145691 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "56a6d526310d478cb4fce34a44f0d197"
format_stamp: "Formatted at 2026-08-06 03:46:34 on dist-test-slave-10pc"
server_key: "2ba0cecd43c4ba11648394aeb9bb4567"
server_key_iv: "37b41625b7517967b8be8d40036e9a19"
server_key_version: "encryptionkey@0"
I20260806 03:46:34.147081 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:46:34.147908 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:34.148097 20956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:46:34.148208 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "56a6d526310d478cb4fce34a44f0d197"
format_stamp: "Formatted at 2026-08-06 03:46:34 on dist-test-slave-10pc"
server_key: "2ba0cecd43c4ba11648394aeb9bb4567"
server_key_iv: "37b41625b7517967b8be8d40036e9a19"
server_key_version: "encryptionkey@0"
I20260806 03:46:34.148293 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:34.151788 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:34.154300 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:34.154470 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:34.158823 20956 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:39219
I20260806 03:46:34.158866 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:39219 every 8 connection(s)
I20260806 03:46:34.159224 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:34.159837 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 20956
I20260806 03:46:34.160002 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:34.160249 30816 external_mini_cluster.cc:1428] Setting key 018ae4e769ee903b4ea9be8493916f4d
I20260806 03:46:34.160622 21024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:34.162875 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:34.160803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58841 (local address 127.30.24.62:39219)
0806 03:46:34.161014 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:34.161025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:34.161057 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:46:34.161510 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:34.161517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:34.161542 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:34.161691 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:34.162420 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:34.162424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:34.162424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:34.162428 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:34.162479 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:34.162489 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:34.162559 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:34.162628 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:46:34.162671 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260806 03:46:34.164029 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:34.164088 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197: Bootstrap starting.
I20260806 03:46:34.164775 21024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:34.165151 21024 log.cc:824] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:34.165994 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197: No bootstrap required, opened a new log
I20260806 03:46:34.166757 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:34.165632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32805 (local address 127.30.24.62:39219)
0806 03:46:34.165672 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:34.165674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:34.165761 (+ 87us) server_negotiation.cc:407] Connection header received
0806 03:46:34.165829 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:34.165829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:34.165834 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:34.165882 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:34.166534 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:34.166534 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:34.166535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:46:34.166537 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:34.166549 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:46:34.166552 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:34.166594 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:34.166651 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 03:46:34.166661 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:46:34.168375 21024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a6d526310d478cb4fce34a44f0d197" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39219 } }
I20260806 03:46:34.168606 21024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:46:34.168663 21024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a6d526310d478cb4fce34a44f0d197, State: Initialized, Role: FOLLOWER
I20260806 03:46:34.168797 21024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a6d526310d478cb4fce34a44f0d197" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39219 } }
I20260806 03:46:34.168895 21024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:46:34.168938 21024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:46:34.168987 21024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:34.169569 21024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a6d526310d478cb4fce34a44f0d197" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39219 } }
I20260806 03:46:34.169695 21024 leader_election.cc:302] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56a6d526310d478cb4fce34a44f0d197; no voters:
I20260806 03:46:34.169868 21024 leader_election.cc:288] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:46:34.169909 21035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:34.170089 21035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [term 1 LEADER]: Becoming Leader. State: Replica: 56a6d526310d478cb4fce34a44f0d197, State: Running, Role: LEADER
I20260806 03:46:34.170142 21024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:34.170230 21035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a6d526310d478cb4fce34a44f0d197" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39219 } }
I20260806 03:46:34.170647 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56a6d526310d478cb4fce34a44f0d197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a6d526310d478cb4fce34a44f0d197" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39219 } } }
I20260806 03:46:34.170776 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:34.170647 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56a6d526310d478cb4fce34a44f0d197. Latest consensus state: current_term: 1 leader_uuid: "56a6d526310d478cb4fce34a44f0d197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a6d526310d478cb4fce34a44f0d197" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 39219 } } }
I20260806 03:46:34.171136 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:34.171278 21043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:34.171800 21043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:34.173434 21043 catalog_manager.cc:1347] Generated new cluster ID: a1d3b93eec2d4f178bcafe0c4c2b546b
I20260806 03:46:34.173492 21043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:34.188756 21043 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:34.188969 21043 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:34.194905 21043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56a6d526310d478cb4fce34a44f0d197: Generated new TSK 0
I20260806 03:46:34.195191 21043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:46:34.199374 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 20956
2026-08-06T03:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1364 ms)
[----------] 18 tests from AuthTokenIssuingTest (24461 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:34 dist-test-slave-10pc krb5kdc[21056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:34 dist-test-slave-10pc krb5kdc[21056](info): set up 2 sockets
Aug 06 03:46:34 dist-test-slave-10pc krb5kdc[21056](info): commencing operation
krb5kdc: starting...
W20260806 03:46:35.254288 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:35.384155 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:36697
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:36697,127.30.24.61:39071,127.30.24.60:44363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:35.468750 21070 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:46:35.468964 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:35.468982 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:35.469009 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:35.470140 21070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:35.470186 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:35.470201 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:35.470216 21070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:35.470227 21070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:35.471746 21070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:36697,127.30.24.61:39071,127.30.24.60:44363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:36697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:35.472023 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:35.472258 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:35.477216 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:35.477360 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:35.477424 21070 server_base.cc:1034] running on GCE node
W20260806 03:46:35.477216 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:35.477775 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:35.478030 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:35.479199 21070 hybrid_clock.cc:648] HybridClock initialized: now 1785987995479159 us; error 55 us; skew 500 ppm
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:35.482069 21070 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:35.482576 21070 webserver.cc:460] Webserver started at http://127.30.24.62:46375/ using document root <none> and password file <none>
I20260806 03:46:35.482836 21070 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:35.482910 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:35.483063 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:35.484151 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "a260fd5cdc954d0a830225d2c0a8110f"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "2bb4cfab32c21267282ff9627986c96a"
server_key_iv: "d3e97a5845ec0ad651c157c9971b8afc"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.484616 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "a260fd5cdc954d0a830225d2c0a8110f"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "2bb4cfab32c21267282ff9627986c96a"
server_key_iv: "d3e97a5845ec0ad651c157c9971b8afc"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.486032 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:35.486827 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:35.487020 21070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 03:46:35.487124 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "a260fd5cdc954d0a830225d2c0a8110f"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "2bb4cfab32c21267282ff9627986c96a"
server_key_iv: "d3e97a5845ec0ad651c157c9971b8afc"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.487210 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:35.500934 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:35.503053 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:35.503223 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:35.507390 21070 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:36697
I20260806 03:46:35.507445 21137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:36697 every 8 connection(s)
I20260806 03:46:35.507848 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:35.508941 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:35.509677 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 21070
I20260806 03:46:35.509804 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
I20260806 03:46:35.509994 30816 external_mini_cluster.cc:1428] Setting key 019ee58118e8384d0205d34853ace340
I20260806 03:46:35.511263 21138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.24.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.61@KRBTEST.COM" created.
I20260806 03:46:35.522571 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.512174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36697 (local address 127.0.0.1:47112)
0806 03:46:35.512376 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.512421 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.512945 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.513254 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.513257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.513498 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.513664 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.513667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.515361 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.515363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.516060 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.516063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.516870 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.516872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.516933 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.517495 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.517504 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.518927 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.520921 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.520928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.520930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.521122 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.521365 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.521366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.521366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.521464 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.521961 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.521967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.522035 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.522139 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.522391 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":98,"thread_start_us":102,"threads_started":1}
I20260806 03:46:35.522574 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.512251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.30.24.62:36697)
0806 03:46:35.512421 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.512430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.512562 (+ 132us) server_negotiation.cc:407] Connection header received
0806 03:46:35.513110 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.513113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.513135 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.513236 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.513788 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.515234 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.516160 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.516742 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.516865 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.519019 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.519029 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.519030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.519039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.520810 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.521199 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.521202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.521202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.521281 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.521536 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.521537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.521538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.521879 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.521982 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.522187 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.522323 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260806 03:46:35.523504 21138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:35.523880 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.523760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55500)
0806 03:46:35.523810 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.523855 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 03:46:35.524045 21088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
I20260806 03:46:35.524555 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.524419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55502)
0806 03:46:35.524493 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.524523 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 03:46:35.524689 21138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:39071: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab.
Entry for principal kudu/127.30.24.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab.
I20260806 03:46:35.539420 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:39071
--webserver_interface=127.30.24.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab
--principal=kudu/127.30.24.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:36697,127.30.24.61:39071,127.30.24.60:44363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:46:35.573954 21138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } attempt: 1
I20260806 03:46:35.574558 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.574391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55510)
0806 03:46:35.574468 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.574521 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 03:46:35.574952 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.574856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55516)
0806 03:46:35.574904 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.574932 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 03:46:35.575138 21138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:39071: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
W20260806 03:46:35.624131 21146 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:46:35.624349 21146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:35.624366 21146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:35.624385 21146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:35.625563 21146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:35.625607 21146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:35.625622 21146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:35.625634 21146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:35.625685 21146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:35.627136 21146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:36697,127.30.24.61:39071,127.30.24.60:44363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab
--principal=kudu/127.30.24.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.24.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:35.627395 21146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:35.627627 21146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:35.632723 21151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:35.632771 21154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:35.632915 21152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:35.632992 21146 server_base.cc:1034] running on GCE node
I20260806 03:46:35.633277 21146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:35.633566 21146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:35.634737 21146 hybrid_clock.cc:648] HybridClock initialized: now 1785987995634722 us; error 46 us; skew 500 ppm
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:35.637297 21146 init.cc:377] Logged in from keytab as kudu/127.30.24.61@KRBTEST.COM (short username kudu)
I20260806 03:46:35.637785 21146 webserver.cc:460] Webserver started at http://127.30.24.61:42601/ using document root <none> and password file <none>
I20260806 03:46:35.638022 21146 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:35.638093 21146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:35.638257 21146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:35.639364 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data/instance:
uuid: "43121f1d0e764c19ab58b97183f460a1"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "e7a85fa8c90304ca7603f5ed53a74dc4"
server_key_iv: "933c06e820f252680d3f22613bc6a0fa"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.639753 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal/instance:
uuid: "43121f1d0e764c19ab58b97183f460a1"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "e7a85fa8c90304ca7603f5ed53a74dc4"
server_key_iv: "933c06e820f252680d3f22613bc6a0fa"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.641546 21146 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 03:46:35.642340 21161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:35.642529 21146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:35.642637 21146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal
uuid: "43121f1d0e764c19ab58b97183f460a1"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "e7a85fa8c90304ca7603f5ed53a74dc4"
server_key_iv: "933c06e820f252680d3f22613bc6a0fa"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.642731 21146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:35.643323 21138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } attempt: 2
I20260806 03:46:35.643744 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.643593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55530)
0806 03:46:35.643659 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.643703 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 03:46:35.644121 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.644024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55536)
0806 03:46:35.644069 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.644101 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 03:46:35.644241 21138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:39071: Network error: Client connection negotiation failed: client connection to 127.30.24.61:39071: connect: Connection refused (error 111)
I20260806 03:46:35.653934 21146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:35.656045 21146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:35.656242 21146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:35.660538 21146 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.61:39071
I20260806 03:46:35.660555 21213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.61:39071 every 8 connection(s)
I20260806 03:46:35.660972 21146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
I20260806 03:46:35.661948 21214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:35.663503 21214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:35.665252 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 21146
I20260806 03:46:35.665359 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-1/wal/instance
I20260806 03:46:35.665555 30816 external_mini_cluster.cc:1428] Setting key cd827582e3292ee05c29dfc7798d67ee
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:35.673069 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.664220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.30.24.62:36697)
0806 03:46:35.664313 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.664316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.664497 (+ 181us) server_negotiation.cc:407] Connection header received
0806 03:46:35.665051 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.665053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.665119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.665186 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.665956 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.666445 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.667443 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.667661 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.667739 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.670166 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.670167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.670167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.670175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.671602 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.672023 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.672024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.672025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.672095 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.672354 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.672355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.672356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.672521 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.672601 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.672886 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.672974 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 03:46:35.673254 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.664120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36697 (local address 127.0.0.1:47126)
0806 03:46:35.664346 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.664391 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.664921 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.665266 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.665270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.665611 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.665856 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.665861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.666545 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.666546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.667346 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.667348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.667764 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.667765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.667907 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.668483 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.668492 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.670065 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.671705 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.671713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.671715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.671942 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.672165 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.672166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.672167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.672278 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.672597 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.672605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.672746 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.672854 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.673017 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.30.24.60@KRBTEST.COM; defaulting to no policy
I20260806 03:46:35.674085 21214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.60@KRBTEST.COM" created.
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:46:35.681577 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.674366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55548)
0806 03:46:35.674417 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.674429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.674472 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.674783 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.674784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.674926 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.674972 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.674974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.675868 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.675869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.676445 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.676447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.677086 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.677087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.677156 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.677606 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.677610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.678778 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.680449 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.680473 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.680474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.680643 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.680879 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.680880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.680881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.680936 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.681246 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.681247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.681287 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.681352 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.681456 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 03:46:35.681653 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.674463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.30.24.61:39071)
0806 03:46:35.674648 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.674652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.674667 (+ 15us) server_negotiation.cc:407] Connection header received
0806 03:46:35.674685 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.674687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.674697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.674783 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.675055 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.675782 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.676557 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.676986 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.677059 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.678882 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.678891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.678892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.678900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.680359 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.680746 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.680749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.680752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.680807 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.681008 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.681009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.681010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.681158 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.681270 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.681388 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.681527 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260806 03:46:35.682344 21214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:35.682734 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.682595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35126)
0806 03:46:35.682644 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.682690 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 03:46:35.682909 21162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
I20260806 03:46:35.683427 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.683296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35130)
0806 03:46:35.683375 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.683406 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 03:46:35.683578 21214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:44363: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab.
Entry for principal kudu/127.30.24.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab.
I20260806 03:46:35.693423 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:44363
--webserver_interface=127.30.24.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab
--principal=kudu/127.30.24.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:36697,127.30.24.61:39071,127.30.24.60:44363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:46:35.732815 21214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } attempt: 1
I20260806 03:46:35.733381 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.733197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35140)
0806 03:46:35.733280 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.733334 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 03:46:35.733774 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.733679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35142)
0806 03:46:35.733725 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.733754 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 03:46:35.733901 21214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:44363: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
I20260806 03:46:35.735409 21138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } attempt: 3
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:46:35.742424 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.735705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55562)
0806 03:46:35.735774 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.735787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.735843 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.736019 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.736020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.736188 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.736266 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.736268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.737001 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.737002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.737561 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.737563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.737975 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.737976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.738053 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.738526 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.738530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.739740 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.741494 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.741495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.741496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.741663 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.741867 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.741867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.741868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.741920 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.742186 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.742187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.742237 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.742287 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.742359 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 03:46:35.742494 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.735768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.30.24.61:39071)
0806 03:46:35.735814 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.735816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.735827 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:35.735924 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.735925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.735939 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.736010 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.736351 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.736915 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.737654 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.737887 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.737964 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.739828 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.739829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.739830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.739837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.741401 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.741733 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.741734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.741734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.741799 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.741981 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.741982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.741982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.742125 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.742191 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.742331 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.742409 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:46:35.742914 21138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:35.743251 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.743139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35146)
0806 03:46:35.743186 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.743218 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 03:46:35.743613 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.743533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35148)
0806 03:46:35.743573 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.743597 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 03:46:35.743723 21138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:44363: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
I20260806 03:46:35.774904 21138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } attempt: 1
I20260806 03:46:35.775489 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.775322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35164)
0806 03:46:35.775395 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.775446 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 03:46:35.775887 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.775784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35178)
0806 03:46:35.775832 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.775863 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 03:46:35.775992 21138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:44363: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
W20260806 03:46:35.779922 21221 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:46:35.780118 21221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:35.780135 21221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:35.780155 21221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:35.781296 21221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:35.781335 21221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:35.781349 21221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:35.781363 21221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:35.781374 21221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:35.782779 21221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:36697,127.30.24.61:39071,127.30.24.60:44363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab
--principal=kudu/127.30.24.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:44363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.24.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:35.782974 21221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:35.783130 21221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:35.787861 21230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:35.787833 21228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:35.787866 21221 server_base.cc:1034] running on GCE node
W20260806 03:46:35.787829 21227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:35.788323 21221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:35.788684 21221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:35.789839 21221 hybrid_clock.cc:648] HybridClock initialized: now 1785987995789825 us; error 30 us; skew 500 ppm
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:35.792289 21221 init.cc:377] Logged in from keytab as kudu/127.30.24.60@KRBTEST.COM (short username kudu)
I20260806 03:46:35.792780 21221 webserver.cc:460] Webserver started at http://127.30.24.60:44629/ using document root <none> and password file <none>
I20260806 03:46:35.793016 21221 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:35.793061 21221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:35.793231 21221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:35.794265 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data/instance:
uuid: "696f2218d5da46398259677c184b9244"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "77f431e5aa2002dd1f9c4e96206af614"
server_key_iv: "c91ca83c1be6bcf987a6e41d30bad83b"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.794636 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal/instance:
uuid: "696f2218d5da46398259677c184b9244"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "77f431e5aa2002dd1f9c4e96206af614"
server_key_iv: "c91ca83c1be6bcf987a6e41d30bad83b"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.795964 21221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 03:46:35.796733 21237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:35.796926 21221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:35.797006 21221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal
uuid: "696f2218d5da46398259677c184b9244"
format_stamp: "Formatted at 2026-08-06 03:46:35 on dist-test-slave-10pc"
server_key: "77f431e5aa2002dd1f9c4e96206af614"
server_key_iv: "c91ca83c1be6bcf987a6e41d30bad83b"
server_key_version: "encryptionkey@0"
I20260806 03:46:35.797060 21221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:35.802069 21214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } attempt: 2
I20260806 03:46:35.802492 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.802352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35192)
0806 03:46:35.802415 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.802456 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 03:46:35.802888 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.802792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35198)
0806 03:46:35.802837 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.802865 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 03:46:35.802999 21214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:44363: Network error: Client connection negotiation failed: client connection to 127.30.24.60:44363: connect: Connection refused (error 111)
I20260806 03:46:35.821084 21221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:35.822983 21221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:35.823096 21221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:35.827091 21221 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.60:44363
I20260806 03:46:35.827131 21289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.60:44363 every 8 connection(s)
I20260806 03:46:35.827513 21221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
I20260806 03:46:35.828613 21290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:35.829093 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 21221
I20260806 03:46:35.829209 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785987724420712-30816-0/minicluster-data/master-2/wal/instance
I20260806 03:46:35.829409 30816 external_mini_cluster.cc:1428] Setting key 5dde1bcf800a28f735b664bc0a40dc3e
I20260806 03:46:35.830641 21290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
I20260806 03:46:35.839550 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.831400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.30.24.62:36697)
0806 03:46:35.831469 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.831472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.831649 (+ 177us) server_negotiation.cc:407] Connection header received
0806 03:46:35.831731 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.831732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.831743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.831792 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.832290 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.832838 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.833678 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.833882 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.833958 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.836075 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.836076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.836076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.836085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.837881 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.838403 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.838404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.838405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.838471 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.838779 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.838779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.838780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.838944 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.839012 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.839347 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.839442 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 03:46:35.839723 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.831328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:36697 (local address 127.0.0.1:47130)
0806 03:46:35.831524 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.831572 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.831667 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.831810 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.831813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.832077 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.832182 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.832187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.832927 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.832928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.833578 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.833581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.833987 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.833989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.834083 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.834598 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.834606 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.835980 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.837967 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.837974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.837976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.838324 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.838547 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.838548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.838548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.838708 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.839015 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.839023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.839197 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.839298 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.839465 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":19,"thread_start_us":130,"threads_started":1}
I20260806 03:46:35.839808 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.830005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.30.24.60:44363)
0806 03:46:35.830188 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.830196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.830222 (+ 26us) server_negotiation.cc:407] Connection header received
0806 03:46:35.830626 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.830632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.830678 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.830794 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.831605 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.832542 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.833305 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.833718 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.833849 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.836651 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.836662 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.836662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.836670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.838438 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.838869 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.838872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.838872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.838924 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.839169 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.839170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.839170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.839339 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.839507 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.839583 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.839672 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260806 03:46:35.840672 21290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:35.847767 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.841004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.30.24.61:39071)
0806 03:46:35.841058 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.841060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.841106 (+ 46us) server_negotiation.cc:407] Connection header received
0806 03:46:35.841172 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.841173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.841183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.841236 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.841516 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.842106 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.842835 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.843091 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.843165 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.845052 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.845053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.845054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.845076 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.846661 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.847029 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.847030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.847030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.847089 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.847251 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.847251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.847252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.847396 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.847466 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.847626 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.847693 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 03:46:35.847785 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.840932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:39071 (local address 127.0.0.1:55564)
0806 03:46:35.841052 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.841065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.841118 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.841241 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.841242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.841390 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.841435 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.841437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.842186 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.842187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.842752 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.842754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.843163 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.843164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.843248 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.843703 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.843707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.844962 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.846721 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.846722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.846723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.846956 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.847154 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.847154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.847155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.847199 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.847462 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.847463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.847498 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.847578 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.847669 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 03:46:35.848373 21290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:35.848920 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.841033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.30.24.62:36697)
0806 03:46:35.841076 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.841078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.841146 (+ 68us) server_negotiation.cc:407] Connection header received
0806 03:46:35.841208 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.841209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.841218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.841271 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.841784 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.842263 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.842920 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.843089 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.843165 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.845799 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.845800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.845800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.845809 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.847496 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.847882 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.847884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.847884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.847940 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.848159 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.848160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.848160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.848299 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.848511 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.848753 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.848824 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 03:46:35.851215 21138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } attempt: 2
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
I20260806 03:46:35.856664 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.848680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35218)
0806 03:46:35.848712 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.848722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.848759 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.848856 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.848856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.849013 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.849044 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.849046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.850104 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.850105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.850559 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.850561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.850945 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.850946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.850995 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.851568 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.851573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.853136 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.854864 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.854865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.854865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.855083 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.855286 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.855287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.855287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.855344 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.856275 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.856277 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.856322 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.856400 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.856525 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260806 03:46:35.856774 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.848714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.30.24.60:44363)
0806 03:46:35.848769 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.848771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.848780 (+ 9us) server_negotiation.cc:407] Connection header received
0806 03:46:35.848801 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.848801 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.848810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.848855 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.849263 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.850020 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.850643 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.850861 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.850936 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.853231 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.853232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.853233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.853239 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.854798 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.855163 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.855163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.855164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.855215 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.855419 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.855420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.855420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.855548 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.855616 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.856479 (+ 863us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.856609 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":41}
I20260806 03:46:35.857673 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.849605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.30.24.61:39071)
0806 03:46:35.849638 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.849639 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.849647 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:35.849697 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.849697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.849705 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.849745 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.850292 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.850723 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.851497 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.851804 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.851870 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.854558 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.854560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.854560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.854567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.856159 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.856567 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.856568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.856569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.856633 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.856847 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.856848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.856848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.856965 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.857120 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.857492 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.857574 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 03:46:35.858876 21290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: Bootstrap starting.
I20260806 03:46:35.859280 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:35.859640 21290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:35.859666 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.851499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35224)
0806 03:46:35.851539 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.851549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.851590 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.851877 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.851878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.852031 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.852064 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.852065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.853374 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.853375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.854253 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.854256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.854736 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.854737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.854805 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.855212 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.855217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.856491 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.858546 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.858547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.858548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.858772 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.859023 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.859024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.859024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.859092 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.859408 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.859409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.859453 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.859497 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.859583 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 03:46:35.859974 21290 log.cc:824] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:35.860107 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.851559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.30.24.60:44363)
0806 03:46:35.851741 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.851743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.851749 (+ 6us) server_negotiation.cc:407] Connection header received
0806 03:46:35.851769 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.851770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.851779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.852160 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.852195 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.853287 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.854360 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.854649 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.854722 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.856603 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.856605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.856605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.856611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.858475 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.858863 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.858864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.858864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.858926 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.859158 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.859159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.859159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.859321 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.859861 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.859907 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.859984 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":125,"threads_started":1}
I20260806 03:46:35.860865 21290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: No bootstrap required, opened a new log
I20260806 03:46:35.860610 21112 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a260fd5cdc954d0a830225d2c0a8110f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47138
I20260806 03:46:35.861740 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f: Bootstrap starting.
I20260806 03:46:35.862411 21138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:35.862398 21290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.862555 21290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696f2218d5da46398259677c184b9244, State: Initialized, Role: FOLLOWER
I20260806 03:46:35.862661 21290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.862716 21138 log.cc:824] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:35.862964 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.863047 21290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:35.863140 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.863580 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f: No bootstrap required, opened a new log
I20260806 03:46:35.864991 21138 raft_consensus.cc:348] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.865219 21138 raft_consensus.cc:729] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a260fd5cdc954d0a830225d2c0a8110f, State: Initialized, Role: FOLLOWER
W20260806 03:46:35.865358 21309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 03:46:35.865353 21138 consensus_queue.cc:260] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
W20260806 03:46:35.865454 21309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:35.865769 21138 sys_catalog.cc:564] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:35.865295 21221 master_runner.cc:186] Error getting master registration for 127.30.24.62:36697: OK, instance_id { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" instance_seqno: 1785987995506247 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:46:35.867374 21221 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:46:35.867447 21112 raft_consensus.cc:482] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:46:35.867494 21112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:46:35.867551 21322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:35.867605 21322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:35.865732 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.867838 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.868261 21112 raft_consensus.cc:504] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.868553 21112 leader_election.cc:288] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [CANDIDATE]: Term 1 election: Requested vote from peers 43121f1d0e764c19ab58b97183f460a1 (127.30.24.61:39071), 696f2218d5da46398259677c184b9244 (127.30.24.60:44363)
I20260806 03:46:35.868893 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a260fd5cdc954d0a830225d2c0a8110f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "696f2218d5da46398259677c184b9244"
I20260806 03:46:35.868999 21264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:35.869836 21264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a260fd5cdc954d0a830225d2c0a8110f in term 1.
I20260806 03:46:35.870069 21086 leader_election.cc:302] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696f2218d5da46398259677c184b9244, a260fd5cdc954d0a830225d2c0a8110f; no voters:
I20260806 03:46:35.870188 21311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:35.870348 21311 raft_consensus.cc:686] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [term 1 LEADER]: Becoming Leader. State: Replica: a260fd5cdc954d0a830225d2c0a8110f, State: Running, Role: LEADER
I20260806 03:46:35.870441 21311 consensus_queue.cc:237] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.870838 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a260fd5cdc954d0a830225d2c0a8110f. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.870919 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:35.871186 21330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:35.871547 21330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:46:35.871449 21070 master_runner.cc:186] Error getting master registration for 127.30.24.61:39071: OK, instance_id { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" instance_seqno: 1785987995659300 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:46:35.873775 21070 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:46:35.882249 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.874163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.30.24.60:44363)
0806 03:46:35.874213 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.874215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.874226 (+ 11us) server_negotiation.cc:407] Connection header received
0806 03:46:35.874372 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.874373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.874385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.874417 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.875588 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.876315 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.877356 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.877699 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.877758 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.879322 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.879323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.879323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.879330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.881025 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.881430 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.881431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.881432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.881499 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.881695 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.881696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.881696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.881820 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.881957 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.882063 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.882138 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 03:46:35.887344 21264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [term 1 FOLLOWER]: Refusing update from remote peer a260fd5cdc954d0a830225d2c0a8110f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:35.887655 21311 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [LEADER]: Connected to new peer: Peer: permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:35.888964 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a260fd5cdc954d0a830225d2c0a8110f. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.889057 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.891510 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.874314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.30.24.62:36697)
0806 03:46:35.874350 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.874352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.882560 (+ 8208us) server_negotiation.cc:407] Connection header received
0806 03:46:35.882679 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.882680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.882691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.882740 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.883417 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.883865 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.884844 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.885150 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.885192 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.886613 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.886614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.886615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.886623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.888629 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.889344 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.889346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.889346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.889413 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.889722 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.889723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.889723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.889854 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.889943 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.891335 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.891414 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:35.892164 21324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.892282 21324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:35.892625 21330 catalog_manager.cc:1347] Generated new cluster ID: 3753cb4c9d034a42a1cd658fd10298ac
I20260806 03:46:35.892724 21330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:35.892872 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.892962 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.893502 21335 mvcc.cc:205] Tried to move back new op lower bound from 7315406831148449792 to 7315406831086014464. Current Snapshot: MvccSnapshot[applied={T|T < 7315406831148449792}]
I20260806 03:46:35.893936 21332 mvcc.cc:205] Tried to move back new op lower bound from 7315406831148449792 to 7315406831086014464. Current Snapshot: MvccSnapshot[applied={T|T < 7315406831148449792 or (T in {7315406831148449792})}]
I20260806 03:46:35.894140 21214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } attempt: 3
Aug 06 03:46:35 dist-test-slave-10pc krb5kdc[21056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
I20260806 03:46:35.899868 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.875078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.30.24.61:39071)
0806 03:46:35.875130 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.875132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.884779 (+ 9647us) server_negotiation.cc:407] Connection header received
0806 03:46:35.884978 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.884980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.885012 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.885043 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.887031 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.887764 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.889309 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.889634 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.889676 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.891167 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.891169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.891169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.891188 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.893787 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.898941 (+ 5154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.898942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.898943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.898998 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.899231 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.899232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.899232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.899367 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.899453 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.899681 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.899761 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 03:46:35.902165 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.894336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:44363 (local address 127.0.0.1:35244)
0806 03:46:35.894421 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:35.894430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:35.894459 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:35.894700 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:35.894701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:35.894848 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:35.894884 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.894885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.895759 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.895759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.896517 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:35.896518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.897019 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.897020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:35.897083 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.897536 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:35.897541 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:35.898961 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:35.901070 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.901072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.901072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.901287 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.901502 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:35.901503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:35.901504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:35.901570 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:35.901856 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:35.901857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:35.901902 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 03:46:35.901969 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 03:46:35.902062 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 03:46:35.902652 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:35.894518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.30.24.60:44363)
0806 03:46:35.894566 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.894567 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.894574 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:46:35.894614 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.894615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.894623 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.894663 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.894968 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.895666 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.896613 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:35.896910 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:35.896959 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:35.899069 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:35.899070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:35.899071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:35.899077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:35.900988 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.901371 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.901372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.901372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.901429 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:35.901641 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:35.901642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:35.901643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:35.901765 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:35.901826 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:35.902488 (+ 662us) server_negotiation.cc:299] Negotiation successful
0806 03:46:35.902567 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 03:46:35.904366 21214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: Bootstrap starting.
I20260806 03:46:35.905143 21214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:35.905483 21214 log.cc:824] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:35.905869 21330 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:35.906606 21330 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:35.912885 21214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: No bootstrap required, opened a new log
I20260806 03:46:35.914964 21214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.915108 21214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43121f1d0e764c19ab58b97183f460a1, State: Initialized, Role: FOLLOWER
I20260806 03:46:35.915192 21214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } }
I20260806 03:46:35.915567 21188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a260fd5cdc954d0a830225d2c0a8110f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43121f1d0e764c19ab58b97183f460a1"
I20260806 03:46:35.915508 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.915663 21187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:35.916693 21187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [term 1 FOLLOWER]: Refusing update from remote peer a260fd5cdc954d0a830225d2c0a8110f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:35.915550 21214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:35.917348 21324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [LEADER]: Connected to new peer: Peer: permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:35.917483 21188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a260fd5cdc954d0a830225d2c0a8110f in term 1.
I20260806 03:46:35.918428 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.919906 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a260fd5cdc954d0a830225d2c0a8110f. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.920287 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.920650 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.920720 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:35.925151 21356 catalog_manager.cc:1259] Loaded cluster ID: 3753cb4c9d034a42a1cd658fd10298ac
I20260806 03:46:35.925235 21356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: loading cluster ID for follower catalog manager: success
I20260806 03:46:35.926363 21356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: acquiring CA information for follower catalog manager: success
W20260806 03:46:35.926522 21356 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260806 03:46:35.929287 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a260fd5cdc954d0a830225d2c0a8110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260fd5cdc954d0a830225d2c0a8110f" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 36697 } } peers { permanent_uuid: "43121f1d0e764c19ab58b97183f460a1" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 39071 } } peers { permanent_uuid: "696f2218d5da46398259677c184b9244" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 44363 } } }
I20260806 03:46:35.929387 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:35.929553 21330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a260fd5cdc954d0a830225d2c0a8110f: Generated new TSK 0
I20260806 03:46:35.929728 21330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:46:35.958379 21143 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.956995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.30.24.62:36697)
0806 03:46:35.957042 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.957044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.957661 (+ 617us) server_negotiation.cc:407] Connection header received
0806 03:46:35.957772 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.957774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.957785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.957815 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.958293 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47146: BlockingRecv error: recv got EOF from 127.0.0.1:47146 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 03:46:35.959556 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.957381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.30.24.61:39071)
0806 03:46:35.957420 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.957422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.958998 (+ 1576us) server_negotiation.cc:407] Connection header received
0806 03:46:35.959150 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.959151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.959162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.959191 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.959495 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55590: BlockingRecv error: recv got EOF from 127.0.0.1:55590 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 03:46:35.960523 21295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.958817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.30.24.60:44363)
0806 03:46:35.958857 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.958859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.959880 (+ 1021us) server_negotiation.cc:407] Connection header received
0806 03:46:35.960039 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.960040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.960052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.960098 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.960427 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35246: BlockingRecv error: recv got EOF from 127.0.0.1:35246 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260806 03:46:35.962929 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.962190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.30.24.61:39071)
0806 03:46:35.962224 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.962225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.962415 (+ 190us) server_negotiation.cc:407] Connection header received
0806 03:46:35.962520 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.962520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.962538 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.962566 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.962884 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55606: BlockingRecv error: recv got EOF from 127.0.0.1:55606 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260806 03:46:35.966054 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.965364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.30.24.61:39071)
0806 03:46:35.965424 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.965426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.965583 (+ 157us) server_negotiation.cc:407] Connection header received
0806 03:46:35.965676 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.965677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.965686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.965718 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.966013 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55620: BlockingRecv error: recv got EOF from 127.0.0.1:55620 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 03:46:35.970006 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.969273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.30.24.61:39071)
0806 03:46:35.969307 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.969308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.969530 (+ 222us) server_negotiation.cc:407] Connection header received
0806 03:46:35.969626 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.969627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.969638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.969665 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.969964 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55632: BlockingRecv error: recv got EOF from 127.0.0.1:55632 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 03:46:35.976027 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.975255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.30.24.61:39071)
0806 03:46:35.975291 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.975293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.975531 (+ 238us) server_negotiation.cc:407] Connection header received
0806 03:46:35.975626 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.975627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.975637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.975666 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.975980 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55636: BlockingRecv error: recv got EOF from 127.0.0.1:55636 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260806 03:46:35.986299 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:35.985372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.30.24.61:39071)
0806 03:46:35.985426 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:35.985428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:35.985706 (+ 278us) server_negotiation.cc:407] Connection header received
0806 03:46:35.985809 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:35.985810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:35.985822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:35.985851 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:35.986240 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55644: BlockingRecv error: recv got EOF from 127.0.0.1:55644 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260806 03:46:36.005069 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:36.003730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.30.24.61:39071)
0806 03:46:36.003789 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:36.003791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:36.004370 (+ 579us) server_negotiation.cc:407] Connection header received
0806 03:46:36.004522 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:36.004523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:36.004537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:36.004569 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:36.004988 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55658: BlockingRecv error: recv got EOF from 127.0.0.1:55658 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 03:46:36.039634 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:36.038626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.30.24.61:39071)
0806 03:46:36.038678 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:36.038680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:36.038986 (+ 306us) server_negotiation.cc:407] Connection header received
0806 03:46:36.039108 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:36.039109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:36.039124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:36.039155 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:36.039571 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55666: BlockingRecv error: recv got EOF from 127.0.0.1:55666 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260806 03:46:36.105943 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:36.105221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.30.24.61:39071)
0806 03:46:36.105319 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:36.105321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:36.105401 (+ 80us) server_negotiation.cc:407] Connection header received
0806 03:46:36.105452 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:36.105453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:36.105468 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:36.105535 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:36.105867 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55674: BlockingRecv error: recv got EOF from 127.0.0.1:55674 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 03:46:36.235829 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:36.235126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.30.24.61:39071)
0806 03:46:36.235237 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:36.235238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:36.235253 (+ 15us) server_negotiation.cc:407] Connection header received
0806 03:46:36.235335 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:36.235337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:36.235347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:36.235402 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:36.235754 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55680: BlockingRecv error: recv got EOF from 127.0.0.1:55680 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 03:46:36.493815 21218 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:36.493097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.30.24.61:39071)
0806 03:46:36.493188 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:36.493190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:36.493203 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:46:36.493286 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:36.493287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:36.493298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:36.493377 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:36.493737 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45056: BlockingRecv error: recv got EOF from 127.0.0.1:45056 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 03:46:36.866097 21309 catalog_manager.cc:1259] Loaded cluster ID: 3753cb4c9d034a42a1cd658fd10298ac
I20260806 03:46:36.866182 21309 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: loading cluster ID for follower catalog manager: success
I20260806 03:46:36.867126 21309 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: acquiring CA information for follower catalog manager: success
I20260806 03:46:36.867427 21309 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 696f2218d5da46398259677c184b9244: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:46:36.927184 21356 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43121f1d0e764c19ab58b97183f460a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:46:37.010076 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:37.007099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.30.24.61:39071)
0806 03:46:37.007248 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:37.007250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:37.007321 (+ 71us) server_negotiation.cc:407] Connection header received
0806 03:46:37.007430 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:37.007431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:37.007503 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:37.007589 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:37.007778 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.008296 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.009207 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.009419 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.009493 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:37.009690 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:37.009816 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:37.009910 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:37.009935 (+ 25us) server_negotiation.cc:299] Negotiation successful
0806 03:46:37.009970 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260806 03:46:37.015062 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:37.012374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.30.24.60:44363)
0806 03:46:37.012541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:37.012543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:37.012553 (+ 10us) server_negotiation.cc:407] Connection header received
0806 03:46:37.012622 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:37.012622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:37.012670 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:37.012759 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:37.012890 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.013399 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.014176 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.014383 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.014461 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:37.014652 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:37.014805 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:37.014894 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:37.014936 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 03:46:37.014959 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260806 03:46:37.020285 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:37.017154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.30.24.61:39071)
0806 03:46:37.017202 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:37.017204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:37.017554 (+ 350us) server_negotiation.cc:407] Connection header received
0806 03:46:37.017665 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:37.017666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:37.017678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:37.017706 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:37.017873 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.018595 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.019469 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.019773 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.019812 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:37.020012 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:37.020044 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:37.020093 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:37.020171 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 03:46:37.020201 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:46:37.020390 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:37.016994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.30.24.62:36697)
0806 03:46:37.017214 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:37.017216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:37.017224 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:37.017273 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:37.017273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:37.017331 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:37.017430 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:37.017579 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.018351 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.019456 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.019678 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.019765 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:37.019942 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:37.020056 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:37.020115 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:37.020262 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 03:46:37.020286 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260806 03:46:37.020521 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:37.017120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.30.24.60:44363)
0806 03:46:37.017183 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:37.017185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:37.017376 (+ 191us) server_negotiation.cc:407] Connection header received
0806 03:46:37.017439 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:37.017440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:37.017451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:37.017515 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:37.017671 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.018411 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.019420 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:37.019704 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:37.019764 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:37.019989 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:37.020022 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:37.020086 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:37.020404 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 03:46:37.020427 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 03:46:37.022579 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 21070
I20260806 03:46:37.029810 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 21146
I20260806 03:46:37.036011 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 21221
2026-08-06T03:46:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2837 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:46:37 dist-test-slave-10pc krb5kdc[21441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:46:37 dist-test-slave-10pc krb5kdc[21441](info): set up 2 sockets
Aug 06 03:46:37 dist-test-slave-10pc krb5kdc[21441](info): commencing operation
krb5kdc: starting...
W20260806 03:46:38.081807 30816 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/test-user.keytab.
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.24.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
Entry for principal kudu/127.30.24.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab.
I20260806 03:46:38.211891 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38817
--webserver_interface=127.30.24.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38817,127.30.24.61:36251,127.30.24.60:40095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:46:38.300873 21454 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:46:38.301172 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:38.301213 21454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:38.301255 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:38.302392 21454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:38.302461 21454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:38.302486 21454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:38.302507 21454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:38.302537 21454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:38.304034 21454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38817,127.30.24.61:36251,127.30.24.60:40095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.62.keytab
--principal=kudu/127.30.24.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.62:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.24.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:38.304332 21454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:38.304625 21454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:38.309676 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:38.309756 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:38.309691 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:38.309893 21454 server_base.cc:1034] running on GCE node
I20260806 03:46:38.310145 21454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:38.310463 21454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:38.311632 21454 hybrid_clock.cc:648] HybridClock initialized: now 1785987998311613 us; error 43 us; skew 500 ppm
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:38.319181 21454 init.cc:377] Logged in from keytab as kudu/127.30.24.62@KRBTEST.COM (short username kudu)
I20260806 03:46:38.319680 21454 webserver.cc:460] Webserver started at http://127.30.24.62:36507/ using document root <none> and password file <none>
I20260806 03:46:38.319931 21454 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:38.320003 21454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:38.320159 21454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:38.321391 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data/instance:
uuid: "ad5a6d15483b497f9f04374bc1d6dd37"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "2d7a86c2a602e5851389af2f84e179bb"
server_key_iv: "b9cfc59eecafbca09ff9a3022f4c1c91"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.321790 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal/instance:
uuid: "ad5a6d15483b497f9f04374bc1d6dd37"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "2d7a86c2a602e5851389af2f84e179bb"
server_key_iv: "b9cfc59eecafbca09ff9a3022f4c1c91"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.323228 21454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 03:46:38.324033 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:38.324287 21454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:38.324404 21454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal
uuid: "ad5a6d15483b497f9f04374bc1d6dd37"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "2d7a86c2a602e5851389af2f84e179bb"
server_key_iv: "b9cfc59eecafbca09ff9a3022f4c1c91"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.324554 21454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:38.350669 21454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:38.353317 21454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:38.353499 21454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:38.357986 21454 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.62:38817
I20260806 03:46:38.358018 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.62:38817 every 8 connection(s)
I20260806 03:46:38.358421 21454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/data/info.pb
I20260806 03:46:38.359532 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:38.362128 21522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.368276 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 21454
I20260806 03:46:38.368386 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-0/wal/instance
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:38.368613 30816 external_mini_cluster.cc:1428] Setting key 0750ace88c28cfaf39a38505aecb5391
I20260806 03:46:38.372366 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.362931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38817 (local address 127.0.0.1:39264)
0806 03:46:38.363084 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.363123 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.363569 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.363845 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.363848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.364101 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.364267 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.364271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.365419 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.365420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.366210 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.366212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.366719 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.366720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.366781 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.367287 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.367296 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.369099 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.370821 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.370829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.370832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.371133 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.371381 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.371382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.371382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.371496 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.371872 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.371878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.371969 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.372049 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.372195 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260806 03:46:38.372588 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.363108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.30.24.62:38817)
0806 03:46:38.363245 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.363250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.363290 (+ 40us) server_negotiation.cc:407] Connection header received
0806 03:46:38.363712 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.363714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.363731 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.363811 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.364351 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.365302 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.366299 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.366617 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.366701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.369183 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.369190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.369191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.369197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.370736 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.371231 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.371234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.371235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.371313 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.371563 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.371564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.371564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.371797 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.371884 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.372112 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.372304 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260806 03:46:38.373636 21522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.374042 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.373924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40556)
0806 03:46:38.373968 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.374010 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 03:46:38.374153 21472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
I20260806 03:46:38.374616 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.374517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40568)
0806 03:46:38.374566 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.374591 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 03:46:38.374719 21522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:36251: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.30.24.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab.
Entry for principal kudu/127.30.24.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab.
I20260806 03:46:38.395725 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:36251
--webserver_interface=127.30.24.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab
--principal=kudu/127.30.24.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38817,127.30.24.61:36251,127.30.24.60:40095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:46:38.423967 21522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } attempt: 1
I20260806 03:46:38.424568 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.424397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40572)
0806 03:46:38.424492 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.424533 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 03:46:38.424955 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.424853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40580)
0806 03:46:38.424902 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.424928 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 03:46:38.425071 21522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:36251: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
W20260806 03:46:38.479393 21529 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:46:38.479653 21529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:38.479686 21529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:38.479729 21529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:38.480947 21529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:38.481009 21529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:38.481031 21529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:38.481052 21529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:38.481083 21529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:38.482614 21529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38817,127.30.24.61:36251,127.30.24.60:40095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.61.keytab
--principal=kudu/127.30.24.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.61:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.24.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:38.482908 21529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:38.483150 21529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:38.488667 21538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:38.488660 21535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:38.488660 21536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:38.488694 21529 server_base.cc:1034] running on GCE node
I20260806 03:46:38.489190 21529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:38.489490 21529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:38.490654 21529 hybrid_clock.cc:648] HybridClock initialized: now 1785987998490636 us; error 42 us; skew 500 ppm
I20260806 03:46:38.493218 21522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } attempt: 2
I20260806 03:46:38.493690 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.493532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40588)
0806 03:46:38.493599 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.493642 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:38.494095 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.493975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40596)
0806 03:46:38.494028 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.494064 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 03:46:38.494256 21522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.61:36251: Network error: Client connection negotiation failed: client connection to 127.30.24.61:36251: connect: Connection refused (error 111)
I20260806 03:46:38.494663 21529 init.cc:377] Logged in from keytab as kudu/127.30.24.61@KRBTEST.COM (short username kudu)
I20260806 03:46:38.495155 21529 webserver.cc:460] Webserver started at http://127.30.24.61:43279/ using document root <none> and password file <none>
I20260806 03:46:38.495430 21529 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:38.495509 21529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:38.495711 21529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:38.497287 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data/instance:
uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "46d585e530f8ede0391297d78c144ec3"
server_key_iv: "df4c90bc40f817b345b57c707e3bdc67"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.497800 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal/instance:
uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "46d585e530f8ede0391297d78c144ec3"
server_key_iv: "df4c90bc40f817b345b57c707e3bdc67"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.499589 21529 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 03:46:38.500687 21545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:38.500897 21529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 03:46:38.501056 21529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal
uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "46d585e530f8ede0391297d78c144ec3"
server_key_iv: "df4c90bc40f817b345b57c707e3bdc67"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.501161 21529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:38.530221 21529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:38.532322 21529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:38.532531 21529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:38.536870 21529 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.61:36251
I20260806 03:46:38.536926 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.61:36251 every 8 connection(s)
I20260806 03:46:38.537334 21529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/data/info.pb
I20260806 03:46:38.538389 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:38.539924 21598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.543483 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 21529
I20260806 03:46:38.543628 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-1/wal/instance
I20260806 03:46:38.543834 30816 external_mini_cluster.cc:1428] Setting key 6cffafcf1ad2c7ca1338bdfda63e64e9
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:38.549824 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.540735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.30.24.62:38817)
0806 03:46:38.540804 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.540807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.541045 (+ 238us) server_negotiation.cc:407] Connection header received
0806 03:46:38.541603 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.541604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.541621 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.541692 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.542322 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.542826 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.543909 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.544153 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.544243 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.546697 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.546699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.546699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.546708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.548281 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.548781 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.548782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.548783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.548846 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.549127 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.549128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.549129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.549254 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.549332 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.549635 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.549732 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:46:38.550006 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.540721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38817 (local address 127.0.0.1:39280)
0806 03:46:38.540915 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.540952 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.541480 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.541758 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.541762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.542018 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.542217 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.542220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.542934 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.542935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.543820 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.543822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.544268 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.544270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.544404 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.545010 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.545019 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.546597 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.548371 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.548379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.548381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.548687 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.548919 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.548920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.548920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.549036 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.549322 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.549329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.549478 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.549589 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.549764 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260806 03:46:38.550709 21598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.24.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.24.60@KRBTEST.COM" created.
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:46:38.560395 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.551005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40612)
0806 03:46:38.551055 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.551260 (+ 205us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.551314 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.551448 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.551449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.551594 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.551641 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.551642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.552540 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.552541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.553034 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.553035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.553609 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.553610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.553683 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.554279 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.554284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.556962 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.559094 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.559095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.559096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.559337 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.559607 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.559607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.559608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.559689 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.560152 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.560152 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.560194 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.560249 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.560324 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 03:46:38.560607 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.551122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.30.24.61:36251)
0806 03:46:38.551271 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.551275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.551312 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:46:38.551372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.551374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.551387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.551450 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.551724 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.552414 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.553141 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.553510 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.553611 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.557060 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.557070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.557071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.557080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.558994 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.559446 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.559449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.559449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.559507 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.559765 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.559766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.559766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.559984 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.560143 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.560317 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.560417 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260806 03:46:38.561296 21598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.561674 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.561563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54540)
0806 03:46:38.561611 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.561656 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 03:46:38.561848 21549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
I20260806 03:46:38.562318 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.562201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54546)
0806 03:46:38.562259 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.562289 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 03:46:38.562456 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:40095: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.24.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab.
Entry for principal kudu/127.30.24.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab.
I20260806 03:46:38.573390 30816 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfG8t0l/build/release/bin/kudu
/tmp/dist-test-taskfG8t0l/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:40095
--webserver_interface=127.30.24.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab
--principal=kudu/127.30.24.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.24.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.24.62:38817,127.30.24.61:36251,127.30.24.60:40095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:46:38.585439 21522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } attempt: 3
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:46:38.593067 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.585876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40618)
0806 03:46:38.585979 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.585991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.586045 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.586223 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.586224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.586374 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.586444 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.586445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.587429 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.587430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.587976 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.587978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.588544 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.588545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.588612 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.589107 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.589112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.590351 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.592058 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.592059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.592060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.592224 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.592439 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.592440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.592440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.592542 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.592813 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.592814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.592855 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.592917 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.592993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 03:46:38.593156 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.585961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.30.24.61:36251)
0806 03:46:38.586028 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.586031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.586043 (+ 12us) server_negotiation.cc:407] Connection header received
0806 03:46:38.586128 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.586130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.586143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.586199 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.586514 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.587343 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.588057 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.588441 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.588538 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.590436 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.590438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.590438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.590446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.591978 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.592297 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.592298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.592298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.592362 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.592606 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.592606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.592607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.592748 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.592818 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.592964 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.593050 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 03:46:38.593653 21522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.594015 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.593920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54554)
0806 03:46:38.593967 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.593997 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 03:46:38.594347 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.594259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54564)
0806 03:46:38.594301 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.594331 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 03:46:38.594429 21522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:40095: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
I20260806 03:46:38.611742 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } attempt: 1
I20260806 03:46:38.612304 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.612136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54572)
0806 03:46:38.612208 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.612259 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 03:46:38.612733 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.612601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54584)
0806 03:46:38.612659 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.612698 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 03:46:38.612869 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:40095: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
I20260806 03:46:38.625568 21522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } attempt: 1
I20260806 03:46:38.626027 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.625893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54598)
0806 03:46:38.625959 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.625998 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 03:46:38.626405 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.626307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54604)
0806 03:46:38.626353 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.626378 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 03:46:38.626513 21522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:40095: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
W20260806 03:46:38.661016 21605 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:46:38.661273 21605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:46:38.661303 21605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:46:38.661346 21605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:46:38.662482 21605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:46:38.662547 21605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:46:38.662569 21605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:46:38.662590 21605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:46:38.662628 21605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:46:38.664110 21605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.24.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.24.62:38817,127.30.24.61:36251,127.30.24.60:40095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/krb5kdc/kudu_127.30.24.60.keytab
--principal=kudu/127.30.24.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.24.60:40095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.24.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 02:39:18 UTC on ccycloud.quasar-kgtzdm.root.comops.site
build id 4569539
I20260806 03:46:38.664394 21605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:46:38.664655 21605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:46:38.669574 21613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:38.669597 21611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:46:38.669595 21610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:46:38.669946 21605 server_base.cc:1034] running on GCE node
I20260806 03:46:38.670217 21605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:46:38.670504 21605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:46:38.671669 21605 hybrid_clock.cc:648] HybridClock initialized: now 1785987998671659 us; error 29 us; skew 500 ppm
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:46:38.674147 21605 init.cc:377] Logged in from keytab as kudu/127.30.24.60@KRBTEST.COM (short username kudu)
I20260806 03:46:38.674634 21605 webserver.cc:460] Webserver started at http://127.30.24.60:45807/ using document root <none> and password file <none>
I20260806 03:46:38.674865 21605 fs_manager.cc:359] Metadata directory not provided
I20260806 03:46:38.674932 21605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:46:38.675087 21605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:46:38.676191 21605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data/instance:
uuid: "10add5da45e44846a8d2e44dde76e98c"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "e956e8edea85e5f79660293129f6f05c"
server_key_iv: "894cf319fecac00add647f878f34bddb"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.676640 21605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal/instance:
uuid: "10add5da45e44846a8d2e44dde76e98c"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "e956e8edea85e5f79660293129f6f05c"
server_key_iv: "894cf319fecac00add647f878f34bddb"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.678004 21605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 03:46:38.678794 21620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:38.678952 21605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 03:46:38.679061 21605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data,/tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal
uuid: "10add5da45e44846a8d2e44dde76e98c"
format_stamp: "Formatted at 2026-08-06 03:46:38 on dist-test-slave-10pc"
server_key: "e956e8edea85e5f79660293129f6f05c"
server_key_iv: "894cf319fecac00add647f878f34bddb"
server_key_version: "encryptionkey@0"
I20260806 03:46:38.679140 21605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:46:38.681064 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } attempt: 2
I20260806 03:46:38.681510 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.681361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54606)
0806 03:46:38.681432 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.681474 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 03:46:38.681933 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.681832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54616)
0806 03:46:38.681881 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.681910 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 03:46:38.682052 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.24.60:40095: Network error: Client connection negotiation failed: client connection to 127.30.24.60:40095: connect: Connection refused (error 111)
I20260806 03:46:38.691045 21605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:46:38.693135 21605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:46:38.693280 21605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:46:38.697782 21605 rpc_server.cc:307] RPC server started. Bound to: 127.30.24.60:40095
I20260806 03:46:38.697852 21672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.24.60:40095 every 8 connection(s)
I20260806 03:46:38.698208 21605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/data/info.pb
I20260806 03:46:38.698746 30816 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfG8t0l/build/release/bin/kudu as pid 21605
I20260806 03:46:38.698835 30816 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfG8t0l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785987724420712-30816-0/minicluster-data/master-2/wal/instance
I20260806 03:46:38.699049 30816 external_mini_cluster.cc:1428] Setting key c37cc2c7c0afcfddbc4a031b03dcda76
I20260806 03:46:38.699433 21673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:46:38.701666 21522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } attempt: 2
I20260806 03:46:38.702620 21673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.62@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
I20260806 03:46:38.712387 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.701974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54626)
0806 03:46:38.702035 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.702047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.702090 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.702310 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.702312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.702466 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.702507 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.702509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.703594 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.703595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.704289 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.704292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.705182 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.705183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.705242 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.705826 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.705832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.708037 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.711287 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.711288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.711497 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.711755 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.711756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.711817 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.712077 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.712079 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.712131 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.712181 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.712283 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 03:46:38.712409 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.699738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.30.24.60:40095)
0806 03:46:38.699940 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.699951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.699983 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:46:38.700721 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.700729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.700779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.700890 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.701897 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.703497 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.704680 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.705069 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.705198 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.708213 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.708214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.708214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.708222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.710460 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.710884 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.710888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.710888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.710986 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711233 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.711234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.711234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.711556 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.711738 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.711871 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.712140 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260806 03:46:38.712560 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.703393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.30.24.62:38817)
0806 03:46:38.703439 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.703441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.703612 (+ 171us) server_negotiation.cc:407] Connection header received
0806 03:46:38.703747 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.703748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.703759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.703820 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.704358 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.705121 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.706310 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.706652 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.706722 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.709063 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.709064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.709064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.709072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.710992 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711451 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.711452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.711452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.711515 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711792 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.711793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.711793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.711992 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.712040 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.712287 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.712373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 03:46:38.712821 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.702025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.30.24.60:40095)
0806 03:46:38.702165 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.702168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.702176 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:38.702197 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.702198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.702207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.702300 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.702586 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.703499 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.704409 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.705073 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.705430 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.708154 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.708162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.708162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.708169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.711197 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711599 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.711600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.711600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.711673 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711889 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.711890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.711890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.711999 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.712562 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.712613 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.712707 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":119,"thread_start_us":130,"threads_started":1}
I20260806 03:46:38.713382 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.703324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.62:38817 (local address 127.0.0.1:39288)
0806 03:46:38.703524 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.703561 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.703643 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.703847 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.703850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.704192 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.704266 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.704268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.705212 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.705212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.706229 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.706233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.706805 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.706806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.706887 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.707402 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.707411 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.708969 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.711095 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.711105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.711370 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.711603 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.711603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.711604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.711709 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.712088 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.712095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.712138 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.712228 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.712333 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260806 03:46:38.713958 21673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.714919 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37: Bootstrap starting.
I20260806 03:46:38.715742 21522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:38.716450 21522 log.cc:824] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:38.717949 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37: No bootstrap required, opened a new log
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.62@KRBTEST.COM
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:46:38.720197 21522 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.720366 21522 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad5a6d15483b497f9f04374bc1d6dd37, State: Initialized, Role: FOLLOWER
I20260806 03:46:38.720482 21522 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.720867 21681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.720882 21522 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:38.720971 21681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.722549 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.713472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.30.24.62:38817)
0806 03:46:38.713548 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.713550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.713563 (+ 13us) server_negotiation.cc:407] Connection header received
0806 03:46:38.713638 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.713639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.713650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.713702 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.714306 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.714781 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.715501 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.715807 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.715868 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.718634 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.718636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.718636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.718643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.720720 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.721486 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.721487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.721488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.721550 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.721830 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.721831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.721832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.721993 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.722270 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.722324 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.722424 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:46:38.722955 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.714231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.61:36251 (local address 127.0.0.1:40632)
0806 03:46:38.715550 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.715561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.715606 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.715801 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.715803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.715966 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.716027 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.716029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.716789 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.716790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.717268 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.717269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.717775 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.717777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.717872 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.718297 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.718303 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.719815 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.721784 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.721785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.721785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.721976 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.722208 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.722210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.722210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.722276 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.722569 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.722570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.722617 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.722685 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.722832 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301}
W20260806 03:46:38.723165 21692 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:38.723222 21692 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:38.723351 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.714317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.30.24.61:36251)
0806 03:46:38.714380 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.714383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.715659 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 03:46:38.715704 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.715706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.715720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.715786 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.716111 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.716702 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.717354 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.717679 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.717756 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.719928 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.719929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.719929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.719935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.721722 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.722056 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.722057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.722057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.722132 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.722344 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.722345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.722346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.722478 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.722542 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.723184 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.723268 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 03:46:38.723701 21673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:46:38.724815 21454 master_runner.cc:186] Error getting master registration for 127.30.24.61:36251: OK, instance_id { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" instance_seqno: 1785987998535630 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:46:38.728215 21454 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.60@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.24.61@KRBTEST.COM
I20260806 03:46:38.732949 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.724098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.30.24.61:36251)
0806 03:46:38.724139 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.724142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.724175 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:46:38.724242 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.724243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.724252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.724291 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.724965 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.725669 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.726685 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.726979 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.727036 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.730095 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.730096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.730097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.730104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.731655 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.732013 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.732015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.732015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.732070 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.732333 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.732333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.732334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.732513 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.732674 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.732757 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.732835 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:46:38.733222 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.723954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54640)
0806 03:46:38.723992 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.724003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.724044 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.724312 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.724313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.724528 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.724565 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.724567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.725535 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.725536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.726379 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.726382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.726974 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.726975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.727042 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.727728 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.727734 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.730011 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.732064 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.732065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.732065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.732284 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.732506 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.732507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.732507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.732577 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.732928 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.732929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.732964 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.733024 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.733126 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260806 03:46:38.733490 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.724042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.30.24.60:40095)
0806 03:46:38.724076 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.724078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.724086 (+ 8us) server_negotiation.cc:407] Connection header received
0806 03:46:38.724106 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.724107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.724117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.724169 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.724656 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.725434 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.726479 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.726813 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.726898 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.730121 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.730122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.730122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.730129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.731956 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.732360 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.732360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.732361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.732413 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.732657 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.732658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.732658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.732843 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.732906 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.733335 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.733416 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260806 03:46:38.734577 30816 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:46:38.735111 21673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: Bootstrap starting.
I20260806 03:46:38.735941 21673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:38.736352 21673 log.cc:824] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:38.736761 21496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ad5a6d15483b497f9f04374bc1d6dd37"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39292
I20260806 03:46:38.736940 21496 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:46:38.737006 21496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:38.737290 21673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: No bootstrap required, opened a new log
I20260806 03:46:38.737973 21496 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.738270 21496 leader_election.cc:288] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [CANDIDATE]: Term 1 election: Requested vote from peers 5ce6fe6e00b44ec7988d329e93c5cd3e (127.30.24.61:36251), 10add5da45e44846a8d2e44dde76e98c (127.30.24.60:40095)
I20260806 03:46:38.739286 21673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.739447 21673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10add5da45e44846a8d2e44dde76e98c, State: Initialized, Role: FOLLOWER
I20260806 03:46:38.739544 21673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.739907 21673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:38.739989 21647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "10add5da45e44846a8d2e44dde76e98c"
I20260806 03:46:38.740141 21647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:38.741067 21647 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5a6d15483b497f9f04374bc1d6dd37 in term 1.
I20260806 03:46:38.741320 21473 leader_election.cc:302] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10add5da45e44846a8d2e44dde76e98c, ad5a6d15483b497f9f04374bc1d6dd37; no voters:
I20260806 03:46:38.741470 21681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:46:38.741659 21681 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [term 1 LEADER]: Becoming Leader. State: Replica: ad5a6d15483b497f9f04374bc1d6dd37, State: Running, Role: LEADER
I20260806 03:46:38.741761 21681 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.742230 21681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad5a6d15483b497f9f04374bc1d6dd37. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.742309 21681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:38.742298 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.742569 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.743486 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:46:38.743703 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 03:46:38.745366 21712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:38.745425 21712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:38.744123 21605 master_runner.cc:186] Error getting master registration for 127.30.24.61:36251: OK, instance_id { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" instance_seqno: 1785987998535630 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:46:38.746428 21605 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:46:38.751873 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.739659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.30.24.62:38817)
0806 03:46:38.739714 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.739716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.740661 (+ 945us) server_negotiation.cc:407] Connection header received
0806 03:46:38.740871 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.740872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.740884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.740913 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.742210 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.742936 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.746601 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.746866 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.746943 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.748553 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.748555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.748556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.748563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.750486 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.750918 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.750919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.750920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.750979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.751201 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.751201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.751202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.751338 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.751446 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.751671 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.751755 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 03:46:38.752108 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.739893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.30.24.61:36251)
0806 03:46:38.739930 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.739932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.744098 (+ 4166us) server_negotiation.cc:407] Connection header received
0806 03:46:38.744210 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.744211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.744221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.744249 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.745018 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.745707 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.746719 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.747682 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.747768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.749136 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.749137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.749138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.749144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.750759 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.751219 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.751221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.751221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.751280 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.751464 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.751465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.751466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.751605 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.751688 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.751902 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.751987 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:38.754088 21647 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [term 1 FOLLOWER]: Refusing update from remote peer ad5a6d15483b497f9f04374bc1d6dd37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:38.754437 21713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:38.755897 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad5a6d15483b497f9f04374bc1d6dd37. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.756139 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.757323 21681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.757419 21681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:38.757619 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.757737 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.759016 21715 catalog_manager.cc:1347] Generated new cluster ID: 7acca61710cc4b76aa7bad02bd445664
I20260806 03:46:38.759071 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:46:38.759868 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.742815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.30.24.60:40095)
0806 03:46:38.742859 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.742861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.744554 (+ 1693us) server_negotiation.cc:407] Connection header received
0806 03:46:38.744636 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.744637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.744649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.744711 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.753172 (+ 8461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.753860 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.754809 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.755137 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.755204 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.756823 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.756824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.756824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.756832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.758505 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.758967 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.758969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.758969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.759052 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.759303 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.759304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.759304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.759440 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.759530 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.759706 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.759782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 03:46:38.773214 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } attempt: 3
Aug 06 03:46:38 dist-test-slave-10pc krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785987998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.24.61@KRBTEST.COM for kudu/127.30.24.60@KRBTEST.COM
I20260806 03:46:38.781032 21719 mvcc.cc:205] Tried to move back new op lower bound from 7315406842893819904 to 7315406842847105024. Current Snapshot: MvccSnapshot[applied={T|T < 7315406842893819904}]
I20260806 03:46:38.781481 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.773453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.24.60:40095 (local address 127.0.0.1:54664)
0806 03:46:38.773545 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0806 03:46:38.773555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 03:46:38.773590 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:46:38.773883 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:46:38.773884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:46:38.774057 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:46:38.774105 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.774106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.774984 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.774985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.775743 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:46:38.775745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.776626 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.776627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:46:38.776686 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.777142 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:46:38.777147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:46:38.778599 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:46:38.780359 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.780360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.780360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.780576 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.780808 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:46:38.780809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:46:38.780809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:46:38.780871 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:46:38.781190 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:46:38.781191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:46:38.781246 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:46:38.781291 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 03:46:38.781381 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 03:46:38.782100 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:38.773665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.30.24.60:40095)
0806 03:46:38.773763 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.773765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.773772 (+ 7us) server_negotiation.cc:407] Connection header received
0806 03:46:38.773791 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.773792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.773801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.773840 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.774174 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.774906 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.776190 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:38.776513 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:38.776564 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:38.778699 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:46:38.778700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:46:38.778700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:46:38.778707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:46:38.780276 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.780672 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.780673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.780674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.780741 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:46:38.780936 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:46:38.780937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:46:38.780937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:46:38.781067 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:46:38.781902 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:38.781944 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 03:46:38.782015 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:46:38.783874 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: Bootstrap starting.
I20260806 03:46:38.784590 21598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: Neither blocks nor log segments found. Creating new log.
I20260806 03:46:38.784880 21598 log.cc:824] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:46:38.789505 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: No bootstrap required, opened a new log
I20260806 03:46:38.791381 21598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.791502 21598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fe6e00b44ec7988d329e93c5cd3e, State: Initialized, Role: FOLLOWER
I20260806 03:46:38.791565 21598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } }
I20260806 03:46:38.791890 21598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:46:38.792689 21571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:46:38.793540 21571 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [term 1 FOLLOWER]: Refusing update from remote peer ad5a6d15483b497f9f04374bc1d6dd37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:46:38.793839 21713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:46:38.795256 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e"
I20260806 03:46:38.796554 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.796663 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.796729 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad5a6d15483b497f9f04374bc1d6dd37. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.796779 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.796811 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.796860 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:46:38.801105 21730 mvcc.cc:205] Tried to move back new op lower bound from 7315406842893819904 to 7315406842847105024. Current Snapshot: MvccSnapshot[applied={T|T < 7315406842893819904}]
I20260806 03:46:38.801858 21572 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ad5a6d15483b497f9f04374bc1d6dd37 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260806 03:46:38.802838 21739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:46:38.802955 21739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:46:38.807986 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5a6d15483b497f9f04374bc1d6dd37" member_type: VOTER last_known_addr { host: "127.30.24.62" port: 38817 } } peers { permanent_uuid: "5ce6fe6e00b44ec7988d329e93c5cd3e" member_type: VOTER last_known_addr { host: "127.30.24.61" port: 36251 } } peers { permanent_uuid: "10add5da45e44846a8d2e44dde76e98c" member_type: VOTER last_known_addr { host: "127.30.24.60" port: 40095 } } }
I20260806 03:46:38.808099 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37 [sys.catalog]: This master's current role is: LEADER
I20260806 03:46:38.812544 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:46:38.813349 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:46:38.819983 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad5a6d15483b497f9f04374bc1d6dd37: Generated new TSK 0
I20260806 03:46:38.820168 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:46:38.852161 21525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.850474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.30.24.62:38817)
0806 03:46:38.850520 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.850522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.851483 (+ 961us) server_negotiation.cc:407] Connection header received
0806 03:46:38.851592 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.851593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.851604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.851635 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.852068 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39320: BlockingRecv error: recv got EOF from 127.0.0.1:39320 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260806 03:46:38.853394 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.851180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.30.24.61:36251)
0806 03:46:38.851221 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.851223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.852701 (+ 1478us) server_negotiation.cc:407] Connection header received
0806 03:46:38.852845 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.852846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.852857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.852886 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.853331 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40662: BlockingRecv error: recv got EOF from 127.0.0.1:40662 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260806 03:46:38.854008 21676 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.850839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.30.24.60:40095)
0806 03:46:38.850879 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.850881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.852568 (+ 1687us) server_negotiation.cc:407] Connection header received
0806 03:46:38.852951 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.852952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.852963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.852992 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.853949 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54680: BlockingRecv error: recv got EOF from 127.0.0.1:54680 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 03:46:38.856796 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.855967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.30.24.61:36251)
0806 03:46:38.856002 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.856004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.856256 (+ 252us) server_negotiation.cc:407] Connection header received
0806 03:46:38.856365 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.856366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.856385 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.856413 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.856748 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40666: BlockingRecv error: recv got EOF from 127.0.0.1:40666 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260806 03:46:38.859809 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.859046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.30.24.61:36251)
0806 03:46:38.859079 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.859081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.859333 (+ 252us) server_negotiation.cc:407] Connection header received
0806 03:46:38.859426 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.859427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.859436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.859466 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.859765 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40672: BlockingRecv error: recv got EOF from 127.0.0.1:40672 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260806 03:46:38.863668 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.862926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.30.24.61:36251)
0806 03:46:38.862958 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.862960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.863200 (+ 240us) server_negotiation.cc:407] Connection header received
0806 03:46:38.863294 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.863294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.863304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.863331 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.863623 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40688: BlockingRecv error: recv got EOF from 127.0.0.1:40688 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 03:46:38.869665 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.868869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.30.24.61:36251)
0806 03:46:38.868903 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.868905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.869160 (+ 255us) server_negotiation.cc:407] Connection header received
0806 03:46:38.869260 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.869261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.869271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.869299 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.869611 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40698: BlockingRecv error: recv got EOF from 127.0.0.1:40698 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 03:46:38.879889 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.878998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.30.24.61:36251)
0806 03:46:38.879039 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.879041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.879327 (+ 286us) server_negotiation.cc:407] Connection header received
0806 03:46:38.879432 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.879433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.879444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.879474 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.879830 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40710: BlockingRecv error: recv got EOF from 127.0.0.1:40710 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 03:46:38.898499 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.897485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.30.24.61:36251)
0806 03:46:38.897532 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.897534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.897844 (+ 310us) server_negotiation.cc:407] Connection header received
0806 03:46:38.897958 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.897959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.897972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.898005 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.898436 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40718: BlockingRecv error: recv got EOF from 127.0.0.1:40718 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 03:46:38.958918 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:38.948806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.30.24.61:36251)
0806 03:46:38.948857 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:38.948859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:38.958216 (+ 9357us) server_negotiation.cc:407] Connection header received
0806 03:46:38.958333 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:38.958335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:38.958348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:38.958378 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:38.958849 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40722: BlockingRecv error: recv got EOF from 127.0.0.1:40722 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260806 03:46:39.025180 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:39.024244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.30.24.61:36251)
0806 03:46:39.024323 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.024326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.024506 (+ 180us) server_negotiation.cc:407] Connection header received
0806 03:46:39.024622 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.024623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.024640 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.024737 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:39.025100 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40736: BlockingRecv error: recv got EOF from 127.0.0.1:40736 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 03:46:39.155345 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:39.154639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.30.24.61:36251)
0806 03:46:39.154713 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.154716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.154745 (+ 29us) server_negotiation.cc:407] Connection header received
0806 03:46:39.154823 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.154824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.154834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.154903 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:39.155261 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40746: BlockingRecv error: recv got EOF from 127.0.0.1:40746 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 03:46:39.413532 21602 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:46:39.412708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.30.24.61:36251)
0806 03:46:39.412780 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.412782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.412867 (+ 85us) server_negotiation.cc:407] Connection header received
0806 03:46:39.412951 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.412952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.412965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.413041 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:46:39.413448 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40760: BlockingRecv error: recv got EOF from 127.0.0.1:40760 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:46:39.745915 21712 catalog_manager.cc:1259] Loaded cluster ID: 7acca61710cc4b76aa7bad02bd445664
I20260806 03:46:39.746001 21712 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: loading cluster ID for follower catalog manager: success
I20260806 03:46:39.746774 21712 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: acquiring CA information for follower catalog manager: success
I20260806 03:46:39.747030 21712 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10add5da45e44846a8d2e44dde76e98c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:46:39.803588 21739 catalog_manager.cc:1259] Loaded cluster ID: 7acca61710cc4b76aa7bad02bd445664
I20260806 03:46:39.803694 21739 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: loading cluster ID for follower catalog manager: success
I20260806 03:46:39.804940 21739 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: acquiring CA information for follower catalog manager: success
I20260806 03:46:39.805365 21739 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ce6fe6e00b44ec7988d329e93c5cd3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:46:39.929415 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:39.926614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.30.24.61:36251)
0806 03:46:39.926750 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.926752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.926769 (+ 17us) server_negotiation.cc:407] Connection header received
0806 03:46:39.926879 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.926880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.926944 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.927035 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:39.927216 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.927764 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.928586 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.928791 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.928870 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:39.929056 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:39.929170 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:39.929264 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:39.929288 (+ 24us) server_negotiation.cc:299] Negotiation successful
0806 03:46:39.929310 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260806 03:46:39.933781 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:39.931178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.30.24.60:40095)
0806 03:46:39.931296 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.931298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.931336 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:46:39.931434 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.931435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.931480 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.931565 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:39.931686 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.932190 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.932981 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.933191 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.933269 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:39.933426 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:39.933531 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:39.933619 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:39.933660 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 03:46:39.933683 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260806 03:46:39.938180 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:39.935378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.30.24.61:36251)
0806 03:46:39.935428 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.935430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.935684 (+ 254us) server_negotiation.cc:407] Connection header received
0806 03:46:39.935760 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.935761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.935772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.935828 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:39.935965 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.936583 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.937501 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.937693 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.937744 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:39.937911 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:39.937943 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:39.937996 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:39.938067 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 03:46:39.938088 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 03:46:39.938774 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:39.935436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.30.24.62:38817)
0806 03:46:39.935581 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.935583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.935626 (+ 43us) server_negotiation.cc:407] Connection header received
0806 03:46:39.935720 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.935721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.935774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.935850 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:39.936064 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.936921 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.937800 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.938118 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.938178 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:39.938393 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:39.938486 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:39.938544 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:39.938645 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 03:46:39.938671 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260806 03:46:39.939610 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:46:39.935595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.30.24.60:40095)
0806 03:46:39.935632 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 03:46:39.935634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 03:46:39.935983 (+ 349us) server_negotiation.cc:407] Connection header received
0806 03:46:39.936004 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:46:39.936005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:46:39.936034 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:46:39.936087 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:46:39.936178 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.936919 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.938764 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:46:39.939076 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:46:39.939153 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:46:39.939299 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:46:39.939327 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:46:39.939380 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 03:46:39.939434 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 03:46:39.939450 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 03:46:39.940868 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 21454
I20260806 03:46:39.948702 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 21529
I20260806 03:46:39.954644 30816 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfG8t0l/build/release/bin/kudu with pid 21605
2026-08-06T03:46:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2917 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5755 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275531 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST